pandas-polars-eda
Master high-performance exploratory data analysis (EDA) using Pandas and Polars, missing value profiling, outlier detection, data distribution analysis, correlation matrices, and vectorised pipelines. Trigger when performing EDA, data cleaning, or memory-efficient tabular analysis in Python.
- License 许可
- MIT
- 最近更新
- 2026-10-17
- Stars
- ★ 8
- Forks
- 1
01 前言 Front Matter
- name
- pandas-polars-eda
- metadata
- { "category": "Data Science and Exploratory Analysis" }
- description
- Master high-performance exploratory data analysis (EDA) using Pandas and Polars, missing value profiling, outlier detection, data distribution analysis, correlation matrices, and vectorised pipelines. Trigger when performing EDA, data cleaning, or memory-efficient tabular analysis in Python.
- compatibility
- Python 3.9+, Pandas 2.0+, Polars 0.20+, PyArrow, Matplotlib, Seaborn
03 SKILL.md 正文
Pandas & Polars EDA Skill Guide
This skill provides production standards, high-performance code patterns, memory optimizations, and data hygiene rules for performing Exploratory Data Analysis (EDA) using Pandas and Polars.
1. Engine Comparison: Pandas vs Polars
+-----------------------+---------------------------------------+---------------------------------------+
| Feature | Pandas (2.0+ with PyArrow) | Polars |
+-----------------------+---------------------------------------+---------------------------------------+
| **Execution Engine** | Single-threaded eager execution | Multi-threaded query optimization |
| **Memory Model** | In-memory numpy / arrow backend | Apache Arrow columnar format native |
| **Evaluation Mode** | Eager only | Eager & Lazy evaluation (`lazy()`) |
| **Performance** | Moderate on datasets > 1GB | Extremely fast (10x-30x speedups) |
+-----------------------+---------------------------------------+---------------------------------------+
2. Automated Data Health & Missing Value Profiling
A. Polars Health Check Pipeline
import polars as pl
def profile_polars_dataframe(df: pl.DataFrame) -> pl.DataFrame:
"""Generate comprehensive dataset health report in Polars."""
null_counts = df.null_count()
dtypes = pl.DataFrame({"column": df.columns, "dtype": [str(d) for d in df.dtypes]})
stats = df.describe()
summary = dtypes.with_columns(
null_count=pl.Series([df[col].null_count() for col in df.columns]),
null_percentage=pl.Series([round((df[col].null_count() / df.height) * 100, 2) for col in df.columns]),
n_unique=pl.Series([df[col].n_unique() for col in df.columns]),
)
return summary
# Usage
df = pl.read_parquet("sales_data.parquet")
health_report = profile_polars_dataframe(df)
print(health_report)
B. Pandas PyArrow Data Profiling
import pandas as pd
def profile_pandas_dataframe(df: pd.DataFrame) -> pd.DataFrame:
"""Generate dataset health report utilizing Pandas 2.0+ PyArrow types."""
profile = pd.DataFrame({
"dtype": df.dtypes,
"null_count": df.isna().sum(),
"null_percentage": (df.isna().sum() / len(df) * 100).round(2),
"unique_values": df.nunique(),
"memory_mb": (df.memory_usage(deep=True) / 1024 / 1024).round(2)
})
return profile
# Read parquet with PyArrow backend for memory efficiency
df_pd = pd.read_parquet("sales_data.parquet", engine="pyarrow", dtype_backend="pyarrow")
print(profile_pandas_dataframe(df_pd))
3. High-Performance Exploratory Analysis Pipelines
A. Polars Lazy API Data Aggregation
Always leverage Polars .lazy() API to allow query optimization prior to execution:
import polars as pl
query = (
pl.scan_parquet("transactions/*.parquet")
.filter(pl.col("status") == "COMPLETED")
.filter(pl.col("transaction_date") >= pl.date(2025, 1, 1))
.group_by(["region", "customer_tier"])
.agg([
pl.col("amount").sum().alias("total_revenue"),
pl.col("amount").mean().alias("avg_order_value"),
pl.col("transaction_id").count().alias("transaction_count"),
pl.col("amount").quantile(0.95).alias("p95_amount")
])
.sort("total_revenue", descending=True)
)
# Execute optimized query plan
results_df = query.collect()
print(results_df)
B. Vectorized Outlier Detection (Interquartile Range - IQR)
def detect_outliers_iqr_polars(df: pl.DataFrame, column_name: str) -> pl.DataFrame:
"""Filter outliers using IQR bounds in Polars."""
q25 = df[column_name].quantile(0.25)
q75 = df[column_name].quantile(0.75)
iqr = q75 - q25
lower_bound = q25 - (1.5 * iqr)
upper_bound = q75 + (1.5 * iqr)
outliers = df.filter(
(pl.col(column_name) < lower_bound) | (pl.col(column_name) > upper_bound)
)
print(f"Detected {outliers.height} outliers out of {df.height} rows for '{column_name}'.")
return outliers
4. Visual Correlation & Distribution Analysis
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
def plot_correlation_heatmap(df: pd.DataFrame, output_path: str = "correlation_matrix.png"):
"""Render high-resolution correlation matrix heatmap."""
numeric_df = df.select_dtypes(include=["number"])
corr_matrix = numeric_df.corr(method="spearman")
plt.figure(figsize=(12, 8), dpi=300)
sns.heatmap(
corr_matrix,
annot=True,
fmt=".2f",
cmap="coolwarm",
square=True,
linewidths=0.5,
cbar_kws={"shrink": 0.8}
)
plt.title("Spearman Rank Correlation Heatmap", fontsize=14, fontweight="bold")
plt.tight_layout()
plt.savefig(output_path)
plt.close()
5. Anti-Patterns & Best Practices
| Anti-Pattern | Performance / Memory Penalty | Production Best Practice |
|---|---|---|
Iterating rows using for index, row in df.iterrows(): |
100x-1000x slower execution speed; causes massive CPU bottleneck. | Always use vectorized expressions (pl.col() in Polars or vectorized Pandas methods). |
| Loading massive CSVs into Pandas eagerly | Causes Out-Of-Memory (OOM) crashes on large files. | Use pl.scan_csv() or convert raw files to Parquet format before analysis. |
Using inplace=True in Pandas |
Does not save memory and will be removed in future Pandas versions. | Assign transformed DataFrames explicitly (df = df.drop(...)). |
Ignoring column data types (object vs category/dictionary) |
object dtypes bloat memory usage by up to 10x. |
Cast high-cardinality string columns to Categorical or String Arrow backend. |
04 同仓库更多 Skills
security-audit
Comprehensive security guidance, source code audit, vulnerability assessment, and penetration test review for codebases, APIs, microservices, CLI tools, libraries, and LLM-backed applications. Triggers on requests to audit a codebase, conduct security reviews, find vulnerabilities or flaws, pen-test source code, or when the user asks in Arabic (فحص أمني، تدقيق الكود أمنياً، مراجعة الثغرات، فحص أمان المشروع). Supports Guidance mode (focused review) and Full Audit mode (deterministic 6-phase harness).
malware-analysis-reversing
Production malware analysis workflows, reverse engineering methodologies, isolated sandbox triage, static/dynamic binary analysis, memory forensics, and YARA detection rule authoring.
application-security-pentesting
Production Application Security (AppSec) standards, OWASP Top 10 vulnerability remediation, SAST/DAST pipeline integration, secure code review, threat modeling, and defensive security posture.
oauth2-oidc-security-flows
Implement enterprise OAuth 2.1 and OpenID Connect (OIDC) authorization code flow with Proof Key for Code Exchange (PKCE), state and nonce verification, refresh token rotation, token revocation, and centralized IdP federation (Keycloak, Auth0, Okta). Trigger when building secure authentication, single sign-on (SSO), or third-party delegated authorization.
jwt-session-management-hardening
Harden JWT-based authentication and session lifecycles using asymmetric signing (RS256/EdDSA), JWKS rotation, strict claims validation, Redis-backed sliding window sessions, revocation lists, and secure HttpOnly cookie storage. Trigger when implementing token validation, session revocation, or hardening API authentication.
soc2-compliance-automation
Best practices for implementing SOC 2 Type II compliance controls, continuous evidence collection, infrastructure policy-as-code, and automated security monitoring. Use when automating Trust Services Criteria (Security, Availability, Confidentiality, Processing Integrity, Privacy), audit logging, IAM least privilege, or CI/CD security gates.