[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7B08H65DMFGW81ADS4V2WEA":3,"skill-same:01M7B08H65DMFGW81ADS4V2WEA":52},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":26,"commitSha":27,"skillDirectory":28,"forksCount":13,"repositoryHtmlUrl":29,"repositoryHomepageUrl":6,"ownerHtmlUrl":30,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":31,"install":32,"resources":45},"01M7B08H65DMFGW81ADS4V2WEA","data-autocleaning",null,"Automated data quality and transformation capabilities for Dataform\u002Fdbt\u002FBigQuery pipelines. Processes data sourced from BigQuery or Cloud Storage (GCS), applying best practices for data ingestion, movement, schema mapping, and comprehensive data cleaning.","Krishp091205\u002Fgemini-config","Krishp091205","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F307182548?v=4","01M7AXCYY4NQ9WQ7DCT3R5Q5S4","01M7AXCYYR7QFCQQ6BFRKD93YH",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fconfig\u002Fskills\u002Fdata-autocleaning\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: data-autocleaning\ndescription: Automated data quality and transformation capabilities for Dataform\u002Fdbt\u002FBigQuery\n  pipelines. Processes data sourced from BigQuery or Cloud Storage (GCS), applying\n  best practices for data ingestion, movement, schema mapping, and comprehensive data\n  cleaning.\nlicense: Apache-2.0\nmetadata:\n  version: v1\n  publisher: google\n---\n\n# Data Autocleaning Skill\n\nAutomated data profiling, quality assessment, and transformation for data\nsourced from **BigQuery** or **Google Cloud Storage (GCS)**.\n\n## When to Use\n\n> [!IMPORTANT]\n>\n> You **MUST** use this skill for **ANY** task where the source is BigQuery or\n> GCS — including seemingly simple operations like \"move data\" or \"copy table\".\n\n-   Apply to **all operations** on new and existing sources: copying, moving,\n    appending, ingesting, or extracting data.\n-   Apply to the **source node** specifically, not to subsequent pipeline steps.\n-   **Never skip** Dataplex profiling (Steps 1 and 3). Always use Dataplex —\n    **not** ad-hoc BigQuery profiling.\n\n## Task Execution Workflow\n\n### Step 1: Preliminary Checks (Before Implementation Planning)\n\nPerform these checks **before** generating the `implementation_plan.md`.\n\n1.  **Check Eligibility** — You MUST confirm the source is a BigQuery table or\n    GCS source.\n2.  **Gather Data Profile via Dataplex**:\n\n    -   **GCS sources**: For GCS sources, you MUST create an external table\n        first before running the dataplex scan.\n    -   **Wait for results**: You **MUST NOT** proceed until the Dataplex\n        profile is available, unless user scan approval was denied.\n    -   Use the profile as input for cleansing and schema mapping decisions. The\n        transformations **MUST NOT** be finalized before profile information is\n        available (unless scan was denied).\n    -   **Commands**:\n\n        1.  **Obtain user approval**: Present the `scripts\u002Fdataplex_scanner.py`\n            scan command to the user and obtain explicit approval before\n            executing it. Use the following template to present the command:\n            -   **Command**: `python3 scripts\u002Fdataplex_scanner.py ...` (Fetch\n                full arguments from step 6 below)\n            -   **Summary**: The script automates Dataplex data profiling. It\n                checks table sizes, applies dynamic sampling for large tables\n                (>1M rows) to reduce costs, skips empty tables, executes\n                concurrent scans for multiple tables, and polls for results\n                automatically.\n            -   **Value Add**: Enables deep data analysis (null rates, distinct\n                values, distributions) allowing data-driven cleansing decisions.\n                It helps identify hidden anomalies (garbage values, format\n                variance) to guide accurate transformations and verifies that\n                the cleaning logic resolves them without introducing\n                regressions.\n            -   **Scope**: The approval obtained here covers all executions of\n                this scanner script for this task (including verification\n                steps).\n        2.  Run the `scripts\u002Fdataplex_scanner.py` script located in the same\n            directory as this `SKILL.md` file. This script handles concurrent\n            scan creation, dynamic sampling for large tables, and polling for\n            results. Use --help to learn more.\n        3.  The script will save the full results as JSON files in the specified\n            output directory.\n        4.  **[!IMPORTANT]** The location MUST be a specific Google Cloud region\n            like `us-central1`; multi-regions like `us` are not supported in\n            Dataplex scan.\n        5.  If there are multiple tables to scan, provide them all in the\n            `--tables` argument to run them concurrently.\n        6.  Use the following command template:\n\n            ```bash\n            python3 scripts\u002Fdataplex_scanner.py \\\n              --tables \u003Cproject.dataset.table> \u003Cproject.catalog.namespace.table> \\\n              --location \u003Clocation> \\\n              --output-dir \u003Coutput_dir>\n            ```\n\n            Note: The script accepts table IDs in the format\n            `project.dataset.table` for BigQuery tables and\n            `project.catalog.namespace.table` for BigLake Iceberg tables.\n\n3.  **Fetch Schema & Samples** — Use `bq` commands to fetch schema and sample\n    data for **both** source and destination tables.\n\n### Step 1.5: Implementation Plan Requirements\n\n1.  Your `implementation_plan.md` **MUST** include a **Profiling Evidence**\n    section. **Note**: If scan execution was denied by the user, document the\n    denial reason here instead of Job IDs.\n\n```markdown\n## Profiling Evidence\n- [ ] Dataplex Data Profile Job ID: \u003CJOB_ID>\n- [ ] Profile Result Summary: \u003CBrief summary of key findings, e.g., % nulls, distinct values>\n```\n\n1.  Your `implementation_plan.md` **MUST** include a step to generate cleansing\n    SQL transformations based on the profile output and instructions in Step 2:\n    Generate Transformations.\n2.  Your `implementation_plan.md` **MUST** also reference Step 3 (Quality\n    Review) under its **Verification Plan** section.\n\n> [!CAUTION]\n>\n> **Do not proceed to implementation** until both sections are completed. You\n> MUST ensure that the verification phase only validates that your\n> transformations successfully addressed the anomalies found in Step 1.\n\n### Step 2: Generate Transformations\n\n#### Schema Alignment\n\n-   Match the destination table schema (types and names) if provided.\n-   **Do not** perform column splits, merges, or other schema operations when no\n    destination table is specified.\n\n#### Data Cleaning Rules\n\n-   **Garbage Values**: Drop or convert to `NULL` only for malformed data (e.g.,\n    unparseable dates, zero-length strings for non-nullable integers).\n-   **Unit Normalization**: Standardize measurable units (e.g., `'C'` → `'F'`)\n    to the most common unit. If units are too varied (e.g., `mg`, `liter`),\n    leave as-is.\n-   **Type Conversion**: Use `COALESCE` with `SAFE.PARSE_*` functions for\n    multiple date\u002Ftime\u002Fdatetime\u002Ftimestamp formats. Fetch diverse samples when\n    source data shows high variance.\n\n#### JSON Data Handling\n\n-   **Parsing**: Use `SAFE.PARSE_JSON` to cast JSON strings to `JSON` type.\n    **Never** use deprecated `JSON_EXTRACT_*`.\n-   **Extraction**: Flatten or extract fields **only** if a destination schema\n    requires it.\n-   **Accessors**: Use `JSON_VALUE`, `JSON_QUERY`, `JSON_QUERY_ARRAY`,\n    `JSON_VALUE_ARRAY` without `SAFE.` prefix (they are safe by default).\n-   **Schema mapping**: When a destination schema is provided, extract JSON\n    fields to match target column names and types.\n-   **NULL handling**: If `SAFE.PARSE_JSON` returns NULL, keep the original\n    string and note the invalid JSON in the cleaning summary.\n\n#### Array Data Handling\n\n-   **Unnesting**: Unnest array fields **only** if a destination schema\n    explicitly requires it.\n-   **Type Casting**: Attempt to cast elements to the most appropriate common\n    type.\n-   **CRITICAL**: Filter out `NULL` elements after `SAFE_CAST` (e.g., using\n    `ARRAY`) as BigQuery arrays cannot contain `NULL`s.\n-   **Normalization**: Only trim whitespace on string fields. Make sure to\n    **PRESERVE CASE**. **DO NOT** perform case conversions (e.g., `LOWER()`,\n    `UPPER()`) unless explicitly required.\n-   **Filtering**: Filter out `NULL` values using `ARRAY_FILTER(array_column, e\n    -> e IS NOT NULL)`.\n-   **Deduplication**: Use `ARRAY(SELECT DISTINCT x FROM UNNEST(array_column))`\n    for case-sensitive deduplication.\n-   **Transformations**: Use `ARRAY_TRANSFORM` or `UNNEST`\u002F`ARRAY_AGG` for\n    element-wise changes (e.g., date parsing).\n-   **Restructuring**: Use `UNNEST` to expand to rows, or `ARRAY_AGG` to group\n    rows into an array, as required by the destination schema.\n\n#### STRUCT\u002FRecord Data Handling\n\n-   **Extraction**: Extract fields to top-level columns **only** if the\n    destination schema requires it.\n-   **Type Casting**: Cast each field using `SAFE_CAST` based on the destination\n    schema or inferred profile.\n-   **Normalization**: Only trim whitespace on string fields. Make sure to\n    **PRESERVE CASE**. **DO NOT** perform case conversions (e.g., `LOWER()`,\n    `UPPER()`) unless explicitly required.\n-   **Field Mapping**: Map directly if structures align; use dot notation (e.g.,\n    `struct.field`) to extract; or use `STRUCT()` constructor to group columns.\n-   **Schema Alignment**: Populate missing fields with `NULL` and drop fields\n    not present in the destination schema.\n\n### Step 3: Quality Review & Profiling\n\n> [!IMPORTANT]\n>\n> You **MUST** verify transformations strictly using the protocol below before\n> completing the task. **Never** skip this step. Use **Dataplex profiling only**\n> (unless scan was denied by the user) — not ad-hoc SQL queries.\n\n**Quality review protocol:**\n\n1.  Extract the `SELECT` query containing all generated transformations\n    (autocleaning, schema mapping, JSON extractions).\n2.  Create a **temporary sample output table** (max 1M rows, 1-hour TTL) by\n    running the transformation query.\n3.  Fix any runtime errors and re-run until the query succeeds.\n4.  **Profile the temporary sample output table using Dataplex**:\n    -   **Verify approval**: If scan execution was approved in Step 1, proceed.\n        If approval was DENIED in Step 1, DO NOT run the scanner script and DO\n        NOT ask the user for approval again. Proceed with manual verification\n        using `bq` sample queries to ensure transformations were successful.\n    -   Run the `scripts\u002Fdataplex_scanner.py` script on the temporary table.\n    -   The script will automatically wait for the profile job to finish and\n        save the results as JSON.\n5.  **Compare profiles** (Skip if scans were denied) — Check the new profile\n    against the Step 1 profile for **every transformed column**:\n\n    ```markdown\n    | Anomaly Type | Threshold |\n    | --- | --- |\n    | **NULL increase** | >1% increase compared to source (unless expected) |\n    | **Value range shift** | Unexpected ranges or formats |\n    ```\n\n6.  **Iterate on anomalies** — For each anomaly:\n\n    1.  **Identify**: Query samples where source is `NOT NULL` but transformed\n        value `IS NULL`.\n    2.  **Fix**: Update the transformation logic.\n    3.  **Repeat**: Re-run Step 3 until the anomaly is resolved.\n\n### Step 3.5: Quality Review Evidence Requirements\n\nYour `walkthrough.md` **MUST** include a **Quality Review Profiling Evidence**\nsection. **Note**: If scan execution was denied by the user, document the denial\nreason here instead of Job IDs.\n\n```markdown\n## Quality Review Profiling Evidence\n- [ ] Post-Transformation Dataplex Profile Job ID: \u003CJOB_ID>\n- [ ] Profile Comparison Summary: \u003CDetailed comparison between initial and final profiles per column>\n```\n\n> [!CAUTION]\n>\n> **Do not** conclude the task or ask for user review until this section is\n> filled and the profile comparison is documented.\n\n### Step 4: Documentation\n\nYour `walkthrough.md` must contain a table for each transformation in the\nfollowing format:\n\n```markdown\n| Field | Description |\n| --- | --- |\n| **Destination schema considered** | The target column\u002Ftype being matched |\n| **Issue Detected** | What data quality problem was found |\n| **Transformation Applied** | The SQL logic used to fix it |\n| **Benefit** | Why this transformation improves the data |\n```\n\nInclude a summary of all quality review steps and profiling evidence.\n\n## Definition of Done\n\n-   All source data quality issues are identified and addressed via SQL\n    transformations.\n-   Verification MUST be completed using the **Quality review protocol** and\n    documented with evidence.\n-   The verification step **MUST only** test the changes and should succeed when\n    the sql is executed.\n-   Transformations align with the target schema if provided.\n-   Cleaning summary is provided with clear justification for each\n    transformation.\n-   If skipped, the reason is clearly stated (e.g., ineligible source).\n","# Data Autocleaning Skill\n\nAutomated data profiling, quality assessment, and transformation for data\nsourced from **BigQuery** or **Google Cloud Storage (GCS)**.\n\n## When to Use\n\n> [!IMPORTANT]\n>\n> You **MUST** use this skill for **ANY** task where the source is BigQuery or\n> GCS — including seemingly simple operations like \"move data\" or \"copy table\".\n\n-   Apply to **all operations** on new and existing sources: copying, moving,\n    appending, ingesting, or extracting data.\n-   Apply to the **source node** specifically, not to subsequent pipeline steps.\n-   **Never skip** Dataplex profiling (Steps 1 and 3). Always use Dataplex —\n    **not** ad-hoc BigQuery profiling.\n\n## Task Execution Workflow\n\n### Step 1: Preliminary Checks (Before Implementation Planning)\n\nPerform these checks **before** generating the `implementation_plan.md`.\n\n1.  **Check Eligibility** — You MUST confirm the source is a BigQuery table or\n    GCS source.\n2.  **Gather Data Profile via Dataplex**:\n\n    -   **GCS sources**: For GCS sources, you MUST create an external table\n        first before running the dataplex scan.\n    -   **Wait for results**: You **MUST NOT** proceed until the Dataplex\n        profile is available, unless user scan approval was denied.\n    -   Use the profile as input for cleansing and schema mapping decisions. The\n        transformations **MUST NOT** be finalized before profile information is\n        available (unless scan was denied).\n    -   **Commands**:\n\n        1.  **Obtain user approval**: Present the `scripts\u002Fdataplex_scanner.py`\n            scan command to the user and obtain explicit approval before\n            executing it. Use the following template to present the command:\n            -   **Command**: `python3 scripts\u002Fdataplex_scanner.py ...` (Fetch\n                full arguments from step 6 below)\n            -   **Summary**: The script automates Dataplex data profiling. It\n                checks table sizes, applies dynamic sampling for large tables\n                (>1M rows) to reduce costs, skips empty tables, executes\n                concurrent scans for multiple tables, and polls for results\n                automatically.\n            -   **Value Add**: Enables deep data analysis (null rates, distinct\n                values, distributions) allowing data-driven cleansing decisions.\n                It helps identify hidden anomalies (garbage values, format\n                variance) to guide accurate transformations and verifies that\n                the cleaning logic resolves them without introducing\n                regressions.\n            -   **Scope**: The approval obtained here covers all executions of\n                this scanner script for this task (including verification\n                steps).\n        2.  Run the `scripts\u002Fdataplex_scanner.py` script located in the same\n            directory as this `SKILL.md` file. This script handles concurrent\n            scan creation, dynamic sampling for large tables, and polling for\n            results. Use --help to learn more.\n        3.  The script will save the full results as JSON files in the specified\n            output directory.\n        4.  **[!IMPORTANT]** The location MUST be a specific Google Cloud region\n            like `us-central1`; multi-regions like `us` are not supported in\n            Dataplex scan.\n        5.  If there are multiple tables to scan, provide them all in the\n            `--tables` argument to run them concurrently.\n        6.  Use the following command template:\n\n            ```bash\n            python3 scripts\u002Fdataplex_scanner.py \\\n              --tables \u003Cproject.dataset.table> \u003Cproject.catalog.namespace.table> \\\n              --location \u003Clocation> \\\n              --output-dir \u003Coutput_dir>\n            ```\n\n            Note: The script accepts table IDs in the format\n            `project.dataset.table` for BigQuery tables and\n            `project.catalog.namespace.table` for BigLake Iceberg tables.\n\n3.  **Fetch Schema & Samples** — Use `bq` commands to fetch schema and sample\n    data for **both** source and destination tables.\n\n### Step 1.5: Implementation Plan Requirements\n\n1.  Your `implementation_plan.md` **MUST** include a **Profiling Evidence**\n    section. **Note**: If scan execution was denied by the user, document the\n    denial reason here instead of Job IDs.\n\n```markdown\n## Profiling Evidence\n- [ ] Dataplex Data Profile Job ID: \u003CJOB_ID>\n- [ ] Profile Result Summary: \u003CBrief summary of key findings, e.g., % nulls, distinct values>\n```\n\n1.  Your `implementation_plan.md` **MUST** include a step to generate cleansing\n    SQL transformations based on the profile output and instructions in Step 2:\n    Generate Transformations.\n2.  Your `implementation_plan.md` **MUST** also reference Step 3 (Quality\n    Review) under its **Verification Plan** section.\n\n> [!CAUTION]\n>\n> **Do not proceed to implementation** until both sections are completed. You\n> MUST ensure that the verification phase only validates that your\n> transformations successfully addressed the anomalies found in Step 1.\n\n### Step 2: Generate Transformations\n\n#### Schema Alignment\n\n-   Match the destination table schema (types and names) if provided.\n-   **Do not** perform column splits, merges, or other schema operations when no\n    destination table is specified.\n\n#### Data Cleaning Rules\n\n-   **Garbage Values**: Drop or convert to `NULL` only for malformed data (e.g.,\n    unparseable dates, zero-length strings for non-nullable integers).\n-   **Unit Normalization**: Standardize measurable units (e.g., `'C'` → `'F'`)\n    to the most common unit. If units are too varied (e.g., `mg`, `liter`),\n    leave as-is.\n-   **Type Conversion**: Use `COALESCE` with `SAFE.PARSE_*` functions for\n    multiple date\u002Ftime\u002Fdatetime\u002Ftimestamp formats. Fetch diverse samples when\n    source data shows high variance.\n\n#### JSON Data Handling\n\n-   **Parsing**: Use `SAFE.PARSE_JSON` to cast JSON strings to `JSON` type.\n    **Never** use deprecated `JSON_EXTRACT_*`.\n-   **Extraction**: Flatten or extract fields **only** if a destination schema\n    requires it.\n-   **Accessors**: Use `JSON_VALUE`, `JSON_QUERY`, `JSON_QUERY_ARRAY`,\n    `JSON_VALUE_ARRAY` without `SAFE.` prefix (they are safe by default).\n-   **Schema mapping**: When a destination schema is provided, extract JSON\n    fields to match target column names and types.\n-   **NULL handling**: If `SAFE.PARSE_JSON` returns NULL, keep the original\n    string and note the invalid JSON in the cleaning summary.\n\n#### Array Data Handling\n\n-   **Unnesting**: Unnest array fields **only** if a destination schema\n    explicitly requires it.\n-   **Type Casting**: Attempt to cast elements to the most appropriate common\n    type.\n-   **CRITICAL**: Filter out `NULL` elements after `SAFE_CAST` (e.g., using\n    `ARRAY`) as BigQuery arrays cannot contain `NULL`s.\n-   **Normalization**: Only trim whitespace on string fields. Make sure to\n    **PRESERVE CASE**. **DO NOT** perform case conversions (e.g., `LOWER()`,\n    `UPPER()`) unless explicitly required.\n-   **Filtering**: Filter out `NULL` values using `ARRAY_FILTER(array_column, e\n    -> e IS NOT NULL)`.\n-   **Deduplication**: Use `ARRAY(SELECT DISTINCT x FROM UNNEST(array_column))`\n    for case-sensitive deduplication.\n-   **Transformations**: Use `ARRAY_TRANSFORM` or `UNNEST`\u002F`ARRAY_AGG` for\n    element-wise changes (e.g., date parsing).\n-   **Restructuring**: Use `UNNEST` to expand to rows, or `ARRAY_AGG` to group\n    rows into an array, as required by the destination schema.\n\n#### STRUCT\u002FRecord Data Handling\n\n-   **Extraction**: Extract fields to top-level columns **only** if the\n    destination schema requires it.\n-   **Type Casting**: Cast each field using `SAFE_CAST` based on the destination\n    schema or inferred profile.\n-   **Normalization**: Only trim whitespace on string fields. Make sure to\n    **PRESERVE CASE**. **DO NOT** perform case conversions (e.g., `LOWER()`,\n    `UPPER()`) unless explicitly required.\n-   **Field Mapping**: Map directly if structures align; use dot notation (e.g.,\n    `struct.field`) to extract; or use `STRUCT()` constructor to group columns.\n-   **Schema Alignment**: Populate missing fields with `NULL` and drop fields\n    not present in the destination schema.\n\n### Step 3: Quality Review & Profiling\n\n> [!IMPORTANT]\n>\n> You **MUST** verify transformations strictly using the protocol below before\n> completing the task. **Never** skip this step. Use **Dataplex profiling only**\n> (unless scan was denied by the user) — not ad-hoc SQL queries.\n\n**Quality review protocol:**\n\n1.  Extract the `SELECT` query containing all generated transformations\n    (autocleaning, schema mapping, JSON extractions).\n2.  Create a **temporary sample output table** (max 1M rows, 1-hour TTL) by\n    running the transformation query.\n3.  Fix any runtime errors and re-run until the query succeeds.\n4.  **Profile the temporary sample output table using Dataplex**:\n    -   **Verify approval**: If scan execution was approved in Step 1, proceed.\n        If approval was DENIED in Step 1, DO NOT run the scanner script and DO\n        NOT ask the user for approval again. Proceed with manual verification\n        using `bq` sample queries to ensure transformations were successful.\n    -   Run the `scripts\u002Fdataplex_scanner.py` script on the temporary table.\n    -   The script will automatically wait for the profile job to finish and\n        save the results as JSON.\n5.  **Compare profiles** (Skip if scans were denied) — Check the new profile\n    against the Step 1 profile for **every transformed column**:\n\n    ```markdown\n    | Anomaly Type | Threshold |\n    | --- | --- |\n    | **NULL increase** | >1% increase compared to source (unless expected) |\n    | **Value range shift** | Unexpected ranges or formats |\n    ```\n\n6.  **Iterate on anomalies** — For each anomaly:\n\n    1.  **Identify**: Query samples where source is `NOT NULL` but transformed\n        value `IS NULL`.\n    2.  **Fix**: Update the transformation logic.\n    3.  **Repeat**: Re-run Step 3 until the anomaly is resolved.\n\n### Step 3.5: Quality Review Evidence Requirements\n\nYour `walkthrough.md` **MUST** include a **Quality Review Profiling Evidence**\nsection. **Note**: If scan execution was denied by the user, document the denial\nreason here instead of Job IDs.\n\n```markdown\n## Quality Review Profiling Evidence\n- [ ] Post-Transformation Dataplex Profile Job ID: \u003CJOB_ID>\n- [ ] Profile Comparison Summary: \u003CDetailed comparison between initial and final profiles per column>\n```\n\n> [!CAUTION]\n>\n> **Do not** conclude the task or ask for user review until this section is\n> filled and the profile comparison is documented.\n\n### Step 4: Documentation\n\nYour `walkthrough.md` must contain a table for each transformation in the\nfollowing format:\n\n```markdown\n| Field | Description |\n| --- | --- |\n| **Destination schema considered** | The target column\u002Ftype being matched |\n| **Issue Detected** | What data quality problem was found |\n| **Transformation Applied** | The SQL logic used to fix it |\n| **Benefit** | Why this transformation improves the data |\n```\n\nInclude a summary of all quality review steps and profiling evidence.\n\n## Definition of Done\n\n-   All source data quality issues are identified and addressed via SQL\n    transformations.\n-   Verification MUST be completed using the **Quality review protocol** and\n    documented with evidence.\n-   The verification step **MUST only** test the changes and should succeed when\n    the sql is executed.\n-   Transformations align with the target schema if provided.\n-   Cleaning summary is provided with clear justification for each\n    transformation.\n-   If skipped, the reason is clearly stated (e.g., ineligible source).\n",{"name":5,"license":14,"metadata":23,"description":7},{"version":24,"publisher":25},"v1","google","main","1f317365436456f9c1039c414a2f6426caabc74c","config\u002Fskills\u002Fdata-autocleaning","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config","https:\u002F\u002Fgithub.com\u002FKrishp091205","2026-08-23 02:02:24",{"source":8,"skillName":5,"commands":33,"prompt":43,"zipAvailable":44},[34,37,40],{"tool":35,"command":36},"npx","npx skills add Krishp091205\u002Fgemini-config --skill data-autocleaning",{"tool":38,"command":39},"bunx","bunx skills add Krishp091205\u002Fgemini-config --skill data-autocleaning",{"tool":41,"command":42},"pnpm","pnpm dlx skills add Krishp091205\u002Fgemini-config --skill data-autocleaning","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fconfig\u002Fskills\u002Fdata-autocleaning\u002FSKILL.md",true,[46],{"path":47,"resourceType":48,"resourceTypeDesc":49,"sizeBytes":50,"sourceHtmlUrl":51},"scripts\u002Fdataplex_scanner.py","SCRIPT","脚本",7528,"https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fconfig\u002Fskills\u002Fdata-autocleaning\u002Fscripts\u002Fdataplex_scanner.py",[53,60,67,74,81,88],{"skillNo":54,"name":55,"nameZh":6,"description":56,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":57,"categories":58,"keywords":59,"publishedAt":18},"01M7B0GYVAXB2M99PR4Z3N1WW0","skill-repair","Use this to fix and re-install agent skills that have failed installation.\nThis skill provides the necessary context and permissions to surgically update\nthe `manifest.json` after a fix has been applied.","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fskills\u002Fskill-repair\u002FSKILL.md",[],[],{"skillNo":61,"name":62,"nameZh":6,"description":63,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":64,"categories":65,"keywords":66,"publishedAt":18},"01M7B0GQ2CKFX5P1DTV802XMAB","notebook-guidance","This skill guides the use of Jupyter notebooks for data analysis, exploration, and visualization, particularly with BigQuery. It outlines best practices for notebook execution and validation (supporting both cell-by-cell execution and full notebook generation depending on tool availability), library installation, and structuring notebooks for clarity. It also covers specific rules for data cleaning, plotting, and integrating with BigQuery SQL and machine learning workflows.\nRelevant when any of the following conditions are true:\n  1. The user request involves a data analysis, data exploration, data visualization, or data insights task that requires multiple steps, queries, or visualizations to answer.\n  2. The user explicitly requests a notebook (.ipynb).\n  3. You are creating, editing, or executing cells in a Jupyter notebook.\n  4. You need to query BigQuery from within a notebook. DO NOT use the Python BigQuery client library; instead, you MUST use the `%%bqsql` magics explained in this skill.","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fskills\u002Fnotebook-guidance\u002FSKILL.md",[],[],{"skillNo":68,"name":69,"nameZh":6,"description":70,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":71,"categories":72,"keywords":73,"publishedAt":18},"01M7B0GJ1K1JE2K593JZT4CAY9","ml-best-practices","CRITICAL RULE: You MUST use this skill whenever the task involves any machine learning tasks or data analysis.\nUse this skill if the user's prompt or requirements mention any of the following:\n  * Clustering\n  * Classification\n  * Regression\n  * Time series forecasting\n  * Statistical testing\n  * Model comparison\n  * ML\n  * Data analysis\n\nSQL\u002FBigQuery ML HANDOFF: If the user requires a SQL solution, use this skill to dictate the ANALYSIS STEPS (e.g., markdown analysis cells, visualization logic), but defer to `bigquery` for all SQL syntax.","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fskills\u002Fml-best-practices\u002FSKILL.md",[],[],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":18},"01M7B0GD7FSN72WCSZ16WEDH42","managing-python-dependencies","Ensures proper Python dependency management, avoiding global `pip install` and\nadhering to project-specific tooling.\n\nUse this skill if any of the following are true:\n  1. Attempting to run `pip install {package_name}`.\n  2. Python packages or dependencies need to be added or modified.\n  3. Initiating a new Python project.\n  4. Creating a new notebook, even if just using BigQuery cells.\n  5. Generating Python code that includes `import` statements for third-party libraries.\n  6. Before executing Python scripts via the terminal to ensure the correct virtual environment is active.","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fskills\u002Fmanaging-python-dependencies\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":18},"01M7B0G6CC6JP6QTQGPSTBMJTX","google-cloud-storage-basics","Stores, retrieves, and manages data as objects in Cloud Storage (Google Cloud Storage, or GCS) buckets. Use when you need to interact with Cloud Storage — create or configure buckets, upload, download, stream, or transfer data, organize objects with folders, generate signed URLs, control access (IAM, ACLs, public access prevention), set storage classes and tiering (Standard, Nearline, Coldline, Archive), manage cost and lifecycle, protect data (versioning, encryption\u002FCMEK, retention and Bucket Lock, object holds, soft delete), host static websites, trigger Pub\u002FSub notifications on object changes, mount buckets as a file system (gcsfuse), or optimize storage performance at any scale. Covers the gcloud storage \u002F gsutil CLI, JSON and XML APIs, client libraries, Terraform, and Cloud Storage MCP servers. Don't use for block storage (Persistent Disk), data warehousing\u002Fanalytics (BigQuery), or databases (Cloud SQL, Spanner, Bigtable, Firestore).","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fskills\u002Fgoogle-cloud-storage-basics\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":18},"01M7B0G2DF32KW9A77S99QBZ4Q","gcs-security-assessment","Assesses security posture, evaluates risks, and checks SAIF compliance for Google Cloud Storage buckets or projects. Use when the user requests security scans, vulnerability checks, or SAIF assessments. Don't use when: The user is asking about non-GCS resources (Compute Engine, GKE, etc.), investigating a live production outage, or asking general security questions not tied to a specific project or bucket.","https:\u002F\u002Fgithub.com\u002FKrishp091205\u002Fgemini-config\u002Fblob\u002Fmain\u002Fskills\u002Fgcs-security-assessment\u002FSKILL.md",[],[]]