[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M79CTVMTKDA4479P8M9SN2N4":3,"skill-same:01M79CTVMTKDA4479P8M9SN2N4":51},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":27,"commitSha":28,"skillDirectory":29,"forksCount":30,"repositoryHtmlUrl":31,"repositoryHomepageUrl":32,"ownerHtmlUrl":33,"ownerBlogUrl":34,"ownerDescription":35,"updatedAt":36,"install":37,"resources":50},"01M79CTVMTKDA4479P8M9SN2N4","pandas-polars-eda",null,"Master high-performance exploratory data analysis (EDA) using Pandas and Polars, missing value profiling, outlier detection, data distribution analysis, correlation matrices, and vectorised pipelines. Trigger when performing EDA, data cleaning, or memory-efficient tabular analysis in Python.","hamzabellouch\u002Fagent-skills","hamzabellouch","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F237856042?v=4","01M79BRYW7XAMK3YDWQEMTWGWS","Hamza Bellouch","01M79BRYWXQV3JK2TGVTX8GN2A",8,"MIT","https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FMath\u002FStatistics_and_Exploratory_Data_Analysis\u002Fpandas-polars-eda\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: pandas-polars-eda\nmetadata:\n  category: Data Science and Exploratory Analysis\ndescription: Master high-performance exploratory data analysis (EDA) using Pandas and Polars, missing value profiling, outlier detection, data distribution analysis, correlation matrices, and vectorised pipelines. Trigger when performing EDA, data cleaning, or memory-efficient tabular analysis in Python.\ncompatibility: Python 3.9+, Pandas 2.0+, Polars 0.20+, PyArrow, Matplotlib, Seaborn\n---\n\n# Pandas & Polars EDA Skill Guide\n\nThis skill provides production standards, high-performance code patterns, memory optimizations, and data hygiene rules for performing Exploratory Data Analysis (EDA) using Pandas and Polars.\n\n---\n\n## 1. Engine Comparison: Pandas vs Polars\n\n```text\n+-----------------------+---------------------------------------+---------------------------------------+\n| Feature               | Pandas (2.0+ with PyArrow)             | Polars                                |\n+-----------------------+---------------------------------------+---------------------------------------+\n| **Execution Engine**  | Single-threaded eager execution       | Multi-threaded query optimization     |\n| **Memory Model**      | In-memory numpy \u002F arrow backend       | Apache Arrow columnar format native   |\n| **Evaluation Mode**   | Eager only                            | Eager & Lazy evaluation (`lazy()`)    |\n| **Performance**       | Moderate on datasets > 1GB            | Extremely fast (10x-30x speedups)     |\n+-----------------------+---------------------------------------+---------------------------------------+\n```\n\n---\n\n## 2. Automated Data Health & Missing Value Profiling\n\n### A. Polars Health Check Pipeline\n\n```python\nimport polars as pl\n\ndef profile_polars_dataframe(df: pl.DataFrame) -> pl.DataFrame:\n    \"\"\"Generate comprehensive dataset health report in Polars.\"\"\"\n    null_counts = df.null_count()\n    dtypes = pl.DataFrame({\"column\": df.columns, \"dtype\": [str(d) for d in df.dtypes]})\n    \n    stats = df.describe()\n    \n    summary = dtypes.with_columns(\n        null_count=pl.Series([df[col].null_count() for col in df.columns]),\n        null_percentage=pl.Series([round((df[col].null_count() \u002F df.height) * 100, 2) for col in df.columns]),\n        n_unique=pl.Series([df[col].n_unique() for col in df.columns]),\n    )\n    return summary\n\n# Usage\ndf = pl.read_parquet(\"sales_data.parquet\")\nhealth_report = profile_polars_dataframe(df)\nprint(health_report)\n```\n\n### B. Pandas PyArrow Data Profiling\n\n```python\nimport pandas as pd\n\ndef profile_pandas_dataframe(df: pd.DataFrame) -> pd.DataFrame:\n    \"\"\"Generate dataset health report utilizing Pandas 2.0+ PyArrow types.\"\"\"\n    profile = pd.DataFrame({\n        \"dtype\": df.dtypes,\n        \"null_count\": df.isna().sum(),\n        \"null_percentage\": (df.isna().sum() \u002F len(df) * 100).round(2),\n        \"unique_values\": df.nunique(),\n        \"memory_mb\": (df.memory_usage(deep=True) \u002F 1024 \u002F 1024).round(2)\n    })\n    return profile\n\n# Read parquet with PyArrow backend for memory efficiency\ndf_pd = pd.read_parquet(\"sales_data.parquet\", engine=\"pyarrow\", dtype_backend=\"pyarrow\")\nprint(profile_pandas_dataframe(df_pd))\n```\n\n---\n\n## 3. High-Performance Exploratory Analysis Pipelines\n\n### A. Polars Lazy API Data Aggregation\n\nAlways leverage Polars `.lazy()` API to allow query optimization prior to execution:\n\n```python\nimport polars as pl\n\nquery = (\n    pl.scan_parquet(\"transactions\u002F*.parquet\")\n    .filter(pl.col(\"status\") == \"COMPLETED\")\n    .filter(pl.col(\"transaction_date\") >= pl.date(2025, 1, 1))\n    .group_by([\"region\", \"customer_tier\"])\n    .agg([\n        pl.col(\"amount\").sum().alias(\"total_revenue\"),\n        pl.col(\"amount\").mean().alias(\"avg_order_value\"),\n        pl.col(\"transaction_id\").count().alias(\"transaction_count\"),\n        pl.col(\"amount\").quantile(0.95).alias(\"p95_amount\")\n    ])\n    .sort(\"total_revenue\", descending=True)\n)\n\n# Execute optimized query plan\nresults_df = query.collect()\nprint(results_df)\n```\n\n### B. Vectorized Outlier Detection (Interquartile Range - IQR)\n\n```python\ndef detect_outliers_iqr_polars(df: pl.DataFrame, column_name: str) -> pl.DataFrame:\n    \"\"\"Filter outliers using IQR bounds in Polars.\"\"\"\n    q25 = df[column_name].quantile(0.25)\n    q75 = df[column_name].quantile(0.75)\n    iqr = q75 - q25\n    \n    lower_bound = q25 - (1.5 * iqr)\n    upper_bound = q75 + (1.5 * iqr)\n    \n    outliers = df.filter(\n        (pl.col(column_name) \u003C lower_bound) | (pl.col(column_name) > upper_bound)\n    )\n    print(f\"Detected {outliers.height} outliers out of {df.height} rows for '{column_name}'.\")\n    return outliers\n```\n\n---\n\n## 4. Visual Correlation & Distribution Analysis\n\n```python\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport pandas as pd\n\ndef plot_correlation_heatmap(df: pd.DataFrame, output_path: str = \"correlation_matrix.png\"):\n    \"\"\"Render high-resolution correlation matrix heatmap.\"\"\"\n    numeric_df = df.select_dtypes(include=[\"number\"])\n    corr_matrix = numeric_df.corr(method=\"spearman\")\n\n    plt.figure(figsize=(12, 8), dpi=300)\n    sns.heatmap(\n        corr_matrix,\n        annot=True,\n        fmt=\".2f\",\n        cmap=\"coolwarm\",\n        square=True,\n        linewidths=0.5,\n        cbar_kws={\"shrink\": 0.8}\n    )\n    plt.title(\"Spearman Rank Correlation Heatmap\", fontsize=14, fontweight=\"bold\")\n    plt.tight_layout()\n    plt.savefig(output_path)\n    plt.close()\n```\n\n---\n\n## 5. Anti-Patterns & Best Practices\n\n| Anti-Pattern | Performance \u002F Memory Penalty | Production Best Practice |\n| :--- | :--- | :--- |\n| **Iterating rows using `for index, row in df.iterrows():`** | 100x-1000x slower execution speed; causes massive CPU bottleneck. | Always use vectorized expressions (`pl.col()` in Polars or vectorized Pandas methods). |\n| **Loading massive CSVs into Pandas eagerly** | Causes Out-Of-Memory (OOM) crashes on large files. | Use `pl.scan_csv()` or convert raw files to Parquet format before analysis. |\n| **Using `inplace=True` in Pandas** | Does not save memory and will be removed in future Pandas versions. | Assign transformed DataFrames explicitly (`df = df.drop(...)`). |\n| **Ignoring column data types (`object` vs `category`\u002F`dictionary`)** | `object` dtypes bloat memory usage by up to 10x. | Cast high-cardinality string columns to `Categorical` or `String` Arrow backend. |\n","# Pandas & Polars EDA Skill Guide\n\nThis skill provides production standards, high-performance code patterns, memory optimizations, and data hygiene rules for performing Exploratory Data Analysis (EDA) using Pandas and Polars.\n\n---\n\n## 1. Engine Comparison: Pandas vs Polars\n\n```text\n+-----------------------+---------------------------------------+---------------------------------------+\n| Feature               | Pandas (2.0+ with PyArrow)             | Polars                                |\n+-----------------------+---------------------------------------+---------------------------------------+\n| **Execution Engine**  | Single-threaded eager execution       | Multi-threaded query optimization     |\n| **Memory Model**      | In-memory numpy \u002F arrow backend       | Apache Arrow columnar format native   |\n| **Evaluation Mode**   | Eager only                            | Eager & Lazy evaluation (`lazy()`)    |\n| **Performance**       | Moderate on datasets > 1GB            | Extremely fast (10x-30x speedups)     |\n+-----------------------+---------------------------------------+---------------------------------------+\n```\n\n---\n\n## 2. Automated Data Health & Missing Value Profiling\n\n### A. Polars Health Check Pipeline\n\n```python\nimport polars as pl\n\ndef profile_polars_dataframe(df: pl.DataFrame) -> pl.DataFrame:\n    \"\"\"Generate comprehensive dataset health report in Polars.\"\"\"\n    null_counts = df.null_count()\n    dtypes = pl.DataFrame({\"column\": df.columns, \"dtype\": [str(d) for d in df.dtypes]})\n    \n    stats = df.describe()\n    \n    summary = dtypes.with_columns(\n        null_count=pl.Series([df[col].null_count() for col in df.columns]),\n        null_percentage=pl.Series([round((df[col].null_count() \u002F df.height) * 100, 2) for col in df.columns]),\n        n_unique=pl.Series([df[col].n_unique() for col in df.columns]),\n    )\n    return summary\n\n# Usage\ndf = pl.read_parquet(\"sales_data.parquet\")\nhealth_report = profile_polars_dataframe(df)\nprint(health_report)\n```\n\n### B. Pandas PyArrow Data Profiling\n\n```python\nimport pandas as pd\n\ndef profile_pandas_dataframe(df: pd.DataFrame) -> pd.DataFrame:\n    \"\"\"Generate dataset health report utilizing Pandas 2.0+ PyArrow types.\"\"\"\n    profile = pd.DataFrame({\n        \"dtype\": df.dtypes,\n        \"null_count\": df.isna().sum(),\n        \"null_percentage\": (df.isna().sum() \u002F len(df) * 100).round(2),\n        \"unique_values\": df.nunique(),\n        \"memory_mb\": (df.memory_usage(deep=True) \u002F 1024 \u002F 1024).round(2)\n    })\n    return profile\n\n# Read parquet with PyArrow backend for memory efficiency\ndf_pd = pd.read_parquet(\"sales_data.parquet\", engine=\"pyarrow\", dtype_backend=\"pyarrow\")\nprint(profile_pandas_dataframe(df_pd))\n```\n\n---\n\n## 3. High-Performance Exploratory Analysis Pipelines\n\n### A. Polars Lazy API Data Aggregation\n\nAlways leverage Polars `.lazy()` API to allow query optimization prior to execution:\n\n```python\nimport polars as pl\n\nquery = (\n    pl.scan_parquet(\"transactions\u002F*.parquet\")\n    .filter(pl.col(\"status\") == \"COMPLETED\")\n    .filter(pl.col(\"transaction_date\") >= pl.date(2025, 1, 1))\n    .group_by([\"region\", \"customer_tier\"])\n    .agg([\n        pl.col(\"amount\").sum().alias(\"total_revenue\"),\n        pl.col(\"amount\").mean().alias(\"avg_order_value\"),\n        pl.col(\"transaction_id\").count().alias(\"transaction_count\"),\n        pl.col(\"amount\").quantile(0.95).alias(\"p95_amount\")\n    ])\n    .sort(\"total_revenue\", descending=True)\n)\n\n# Execute optimized query plan\nresults_df = query.collect()\nprint(results_df)\n```\n\n### B. Vectorized Outlier Detection (Interquartile Range - IQR)\n\n```python\ndef detect_outliers_iqr_polars(df: pl.DataFrame, column_name: str) -> pl.DataFrame:\n    \"\"\"Filter outliers using IQR bounds in Polars.\"\"\"\n    q25 = df[column_name].quantile(0.25)\n    q75 = df[column_name].quantile(0.75)\n    iqr = q75 - q25\n    \n    lower_bound = q25 - (1.5 * iqr)\n    upper_bound = q75 + (1.5 * iqr)\n    \n    outliers = df.filter(\n        (pl.col(column_name) \u003C lower_bound) | (pl.col(column_name) > upper_bound)\n    )\n    print(f\"Detected {outliers.height} outliers out of {df.height} rows for '{column_name}'.\")\n    return outliers\n```\n\n---\n\n## 4. Visual Correlation & Distribution Analysis\n\n```python\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport pandas as pd\n\ndef plot_correlation_heatmap(df: pd.DataFrame, output_path: str = \"correlation_matrix.png\"):\n    \"\"\"Render high-resolution correlation matrix heatmap.\"\"\"\n    numeric_df = df.select_dtypes(include=[\"number\"])\n    corr_matrix = numeric_df.corr(method=\"spearman\")\n\n    plt.figure(figsize=(12, 8), dpi=300)\n    sns.heatmap(\n        corr_matrix,\n        annot=True,\n        fmt=\".2f\",\n        cmap=\"coolwarm\",\n        square=True,\n        linewidths=0.5,\n        cbar_kws={\"shrink\": 0.8}\n    )\n    plt.title(\"Spearman Rank Correlation Heatmap\", fontsize=14, fontweight=\"bold\")\n    plt.tight_layout()\n    plt.savefig(output_path)\n    plt.close()\n```\n\n---\n\n## 5. Anti-Patterns & Best Practices\n\n| Anti-Pattern | Performance \u002F Memory Penalty | Production Best Practice |\n| :--- | :--- | :--- |\n| **Iterating rows using `for index, row in df.iterrows():`** | 100x-1000x slower execution speed; causes massive CPU bottleneck. | Always use vectorized expressions (`pl.col()` in Polars or vectorized Pandas methods). |\n| **Loading massive CSVs into Pandas eagerly** | Causes Out-Of-Memory (OOM) crashes on large files. | Use `pl.scan_csv()` or convert raw files to Parquet format before analysis. |\n| **Using `inplace=True` in Pandas** | Does not save memory and will be removed in future Pandas versions. | Assign transformed DataFrames explicitly (`df = df.drop(...)`). |\n| **Ignoring column data types (`object` vs `category`\u002F`dictionary`)** | `object` dtypes bloat memory usage by up to 10x. | Cast high-cardinality string columns to `Categorical` or `String` Arrow backend. |\n",{"name":5,"metadata":24,"description":7,"compatibility":26},{"category":25},"Data Science and Exploratory Analysis","Python 3.9+, Pandas 2.0+, Polars 0.20+, PyArrow, Matplotlib, Seaborn","main","e905af21e5f1e1c3ca605e9656c3a7930360a84e","Math\u002FStatistics_and_Exploratory_Data_Analysis\u002Fpandas-polars-eda",1,"https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills","https:\u002F\u002Fhamzabellouch.github.io\u002F","https:\u002F\u002Fgithub.com\u002Fhamzabellouch","hamzabellouch.github.io","CEO & Founder of @tknoglobal | Researcher in Technology & Applied Sciences.","2026-10-17 03:20:19",{"source":8,"skillName":5,"commands":38,"prompt":48,"zipAvailable":49},[39,42,45],{"tool":40,"command":41},"npx","npx skills add hamzabellouch\u002Fagent-skills --skill pandas-polars-eda",{"tool":43,"command":44},"bunx","bunx skills add hamzabellouch\u002Fagent-skills --skill pandas-polars-eda",{"tool":46,"command":47},"pnpm","pnpm dlx skills add hamzabellouch\u002Fagent-skills --skill pandas-polars-eda","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FMath\u002FStatistics_and_Exploratory_Data_Analysis\u002Fpandas-polars-eda\u002FSKILL.md",true,[],[52,59,66,73,80,87],{"skillNo":53,"name":54,"nameZh":6,"description":55,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":56,"categories":57,"keywords":58,"publishedAt":19},"01M79FHH2H815A9TAP6WNKH4Q8","security-audit","Comprehensive security guidance, source code audit, vulnerability assessment, and penetration test review for codebases, APIs, microservices, CLI tools, libraries, and LLM-backed applications. Triggers on requests to audit a codebase, conduct security reviews, find vulnerabilities or flaws, pen-test source code, or when the user asks in Arabic (فحص أمني، تدقيق الكود أمنياً، مراجعة الثغرات، فحص أمان المشروع). Supports Guidance mode (focused review) and Full Audit mode (deterministic 6-phase harness).","https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FSecurity\u002FSecurity_Audit_and_Code_Review\u002Fsecurity-audit\u002FSKILL.md",[],[],{"skillNo":60,"name":61,"nameZh":6,"description":62,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":63,"categories":64,"keywords":65,"publishedAt":19},"01M79FHBGXVBY6VQ1X7ETAZMF1","malware-analysis-reversing","Production malware analysis workflows, reverse engineering methodologies, isolated sandbox triage, static\u002Fdynamic binary analysis, memory forensics, and YARA detection rule authoring.","https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FSecurity\u002FOffensive_Security_and_Pentesting\u002Fmalware-analysis-reversing\u002FSKILL.md",[],[],{"skillNo":67,"name":68,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":72,"publishedAt":19},"01M79FH6EACDR801TZCE4R8SAC","application-security-pentesting","Production Application Security (AppSec) standards, OWASP Top 10 vulnerability remediation, SAST\u002FDAST pipeline integration, secure code review, threat modeling, and defensive security posture.","https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FSecurity\u002FOffensive_Security_and_Pentesting\u002Fapplication-security-pentesting\u002FSKILL.md",[],[],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":77,"categories":78,"keywords":79,"publishedAt":19},"01M79FH0VKVT44FK923K520ZQA","oauth2-oidc-security-flows","Implement enterprise OAuth 2.1 and OpenID Connect (OIDC) authorization code flow with Proof Key for Code Exchange (PKCE), state and nonce verification, refresh token rotation, token revocation, and centralized IdP federation (Keycloak, Auth0, Okta). Trigger when building secure authentication, single sign-on (SSO), or third-party delegated authorization.","https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FSecurity\u002FIdentity_and_Access_Management_IAM\u002Foauth2-oidc-security-flows\u002FSKILL.md",[],[],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":19},"01M79FGVVB3R4T88XXD8FRSV5B","jwt-session-management-hardening","Harden JWT-based authentication and session lifecycles using asymmetric signing (RS256\u002FEdDSA), JWKS rotation, strict claims validation, Redis-backed sliding window sessions, revocation lists, and secure HttpOnly cookie storage. Trigger when implementing token validation, session revocation, or hardening API authentication.","https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FSecurity\u002FIdentity_and_Access_Management_IAM\u002Fjwt-session-management-hardening\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":19},"01M79FGNCPZ85CVEY0B3NE4H3B","soc2-compliance-automation","Best practices for implementing SOC 2 Type II compliance controls, continuous evidence collection, infrastructure policy-as-code, and automated security monitoring. Use when automating Trust Services Criteria (Security, Availability, Confidentiality, Processing Integrity, Privacy), audit logging, IAM least privilege, or CI\u002FCD security gates.","https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FSecurity\u002FGovernance_Compliance_and_Legal_Tech\u002Fsoc2-compliance-automation\u002FSKILL.md",[],[]]