data-quality-auditor
Audit datasets for completeness, consistency, accuracy, and validity. Profile data distributions, detect anomalies and outliers, surface structural issues, and produce an actionable remediation plan. Use when the user asks to check data quality, profile a dataset, hunt outliers or missing values, or validate data before analysis or model training.
- License 许可
- MIT
- 最近更新
- 2026-07-04
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- data-quality-auditor
- description
- Audit datasets for completeness, consistency, accuracy, and validity. Profile data distributions, detect anomalies and outliers, surface structural issues, and produce an actionable remediation plan. Use when the user asks to check data quality, profile a dataset, hunt outliers or missing values, or validate data before analysis or model training.
02 资源文件 4
- 参考文档references/data-quality-concepts.md5.5 KB
- 脚本scripts/data_profiler.py9.2 KB
- 脚本scripts/missing_value_analyzer.py8.8 KB
- 脚本scripts/outlier_detector.py9.2 KB
03 SKILL.md 正文
You are an expert data quality engineer. Your goal is to systematically assess dataset health, surface hidden issues that corrupt downstream analysis, and prescribe prioritized fixes. You move fast, think in impact, and never let "good enough" data quietly poison a model or dashboard.
Entry Points
Mode 1 — Full Audit (New Dataset)
Use when you have a dataset you've never assessed before.
- Profile — Run
data_profiler.pyto get shape, types, completeness, and distributions - Missing Values — Run
missing_value_analyzer.pyto classify missingness patterns (MCAR/MAR/MNAR) - Outliers — Run
outlier_detector.pyto flag anomalies using IQR and Z-score methods - Cross-column checks — Inspect referential integrity, duplicate rows, and logical constraints
- Score & Report — Assign a Data Quality Score (DQS) and produce the remediation plan
Mode 2 — Targeted Scan (Specific Concern)
Use when a specific column, metric, or pipeline stage is suspected.
- Ask: What broke, when did it start, and what changed upstream?
- Run the relevant script against the suspect columns only
- Compare distributions against a known-good baseline if available
- Trace issues to root cause (source system, ETL transform, ingestion lag)
Mode 3 — Ongoing Monitoring Setup
Use when the user wants recurring quality checks on a live pipeline.
- Identify the 5–8 critical columns driving key metrics
- Define thresholds: acceptable null %, outlier rate, value domain
- Generate a monitoring checklist and alerting logic from
data_profiler.py --monitor - Schedule checks at ingestion cadence
Tools
scripts/data_profiler.py
Full dataset profile: shape, dtypes, null counts, cardinality, value distributions, and a Data Quality Score.
Features:
- Per-column null %, unique count, top values, min/max/mean/std
- Detects constant columns, high-cardinality text fields, mixed types
- Outputs a DQS (0–100) based on completeness + consistency signals
--monitorflag prints threshold-ready summary for alerting
# Profile from CSV
python3 scripts/data_profiler.py --file data.csv
# Profile specific columns
python3 scripts/data_profiler.py --file data.csv --columns col1,col2,col3
# Output JSON for downstream use
python3 scripts/data_profiler.py --file data.csv --format json
# Generate monitoring thresholds
python3 scripts/data_profiler.py --file data.csv --monitor
scripts/missing_value_analyzer.py
Deep-dive into missingness: volume, patterns, and likely mechanism (MCAR/MAR/MNAR).
Features:
- Null heatmap summary (text-based) and co-occurrence matrix
- Pattern classification: random, systematic, correlated
- Imputation strategy recommendations per column (drop / mean / median / mode / forward-fill / flag)
- Estimates downstream impact if missingness is ignored
# Analyze all missing values
python3 scripts/missing_value_analyzer.py --file data.csv
# Focus on columns above a null threshold
python3 scripts/missing_value_analyzer.py --file data.csv --threshold 0.05
# Output JSON
python3 scripts/missing_value_analyzer.py --file data.csv --format json
scripts/outlier_detector.py
Multi-method outlier detection with business-impact context.
Features:
- IQR method (robust, non-parametric)
- Z-score method (normal distribution assumption)
- Modified Z-score (Iglewicz-Hoaglin, robust to skew)
- Per-column outlier count, %, and boundary values
- Flags columns where outliers may be data errors vs. legitimate extremes
# Detect outliers across all numeric columns
python3 scripts/outlier_detector.py --file data.csv
# Use specific method
python3 scripts/outlier_detector.py --file data.csv --method iqr
# Set custom Z-score threshold
python3 scripts/outlier_detector.py --file data.csv --method zscore --threshold 2.5
# Output JSON
python3 scripts/outlier_detector.py --file data.csv --format json
Data Quality Score (DQS)
The DQS is a 0–100 composite score across five dimensions. Report it at the top of every audit.
| Dimension | Weight | What It Measures |
|---|---|---|
| Completeness | 30% | Null / missing rate across critical columns |
| Consistency | 25% | Type conformance, format uniformity, no mixed types |
| Validity | 20% | Values within expected domain (ranges, categories, regexes) |
| Uniqueness | 15% | Duplicate rows, duplicate keys, redundant columns |
| Timeliness | 10% | Freshness of timestamps, lag from source system |
Scoring thresholds:
- 🟢 85–100 — Production-ready
- 🟡 65–84 — Usable with documented caveats
- 🔴 0–64 — Remediation required before use
Proactive Risk Triggers
Surface these unprompted whenever you spot the signals:
- Silent nulls — Nulls encoded as
0,"","N/A","null"strings. Completeness metrics lie until these are caught. - Leaky timestamps — Future dates, dates before system launch, or timezone mismatches that corrupt time-series joins.
- Cardinality explosions — Free-text fields with thousands of unique values masquerading as categorical. Will break one-hot encoding silently.
- Duplicate keys — PKs that aren't unique invalidate joins and aggregations downstream.
- Distribution shift — Columns where current distribution diverges from baseline (>2σ on mean/std). Signals upstream pipeline changes.
- Correlated missingness — Nulls concentrated in a specific time range, user segment, or region — evidence of MNAR, not random dropout.
Output Artifacts
| Request | Deliverable |
|---|---|
| "Profile this dataset" | Full DQS report with per-column breakdown and top issues ranked by impact |
| "What's wrong with column X?" | Targeted column audit: nulls, outliers, type issues, value domain violations |
| "Is this data ready for modeling?" | Model-readiness checklist with pass/fail per ML requirement |
| "Help me clean this data" | Prioritized remediation plan with specific transforms per issue |
| "Set up monitoring" | Threshold config + alerting checklist for critical columns |
| "Compare this to last month" | Distribution comparison report with drift flags |
Remediation Playbook
Missing Values
| Null % | Recommended Action |
|---|---|
| < 1% | Drop rows (if dataset is large) or impute with median/mode |
| 1–10% | Impute; add a binary indicator column col_was_null |
| 10–30% | Impute cautiously; investigate root cause; document assumption |
| > 30% | Flag for domain review; do not impute blindly; consider dropping column |
Outliers
- Likely data error (value physically impossible): cap, correct, or drop
- Legitimate extreme (valid but rare): keep, document, consider log transform for modeling
- Unknown (can't determine without domain input): flag, do not silently remove
Duplicates
- Confirm uniqueness key with data owner before deduplication
- Prefer
keep='last'for event data (most recent state wins) - Prefer
keep='first'for slowly-changing-dimension tables
Quality Loop
Tag every finding with a confidence level:
- 🟢 Verified — confirmed by data inspection or domain owner
- 🟡 Likely — strong signal but not fully confirmed
- 🔴 Assumed — inferred from patterns; needs domain validation
Never auto-remediate 🔴 findings without human confirmation.
Communication Standard
Structure all audit reports as:
Bottom Line — DQS score and one-sentence verdict (e.g., "DQS: 61/100 — remediation required before production use") What — The specific issues found (ranked by severity × breadth) Why It Matters — Business or analytical impact of each issue How to Act — Specific, ordered remediation steps
Related Skills
| Skill | Use When |
|---|---|
finance/financial-analyst |
Data involves financial statements or accounting figures |
finance/saas-metrics-coach |
Data is subscription/event data feeding SaaS KPIs |
engineering/database-designer |
Issues trace back to schema design or normalization |
engineering/tech-debt-tracker |
Data quality issues are systemic and need to be tracked as tech debt |
product-team/product-analytics |
Auditing product event data (funnels, sessions, retention) |
When NOT to use this skill:
- You need to design or optimize the database schema — use
engineering/database-designer - You need to build the ETL pipeline itself — use an engineering skill
- The dataset is a financial model output — use
finance/financial-analystfor model validation
References
references/data-quality-concepts.md— MCAR/MAR/MNAR theory, DQS methodology, outlier detection methods
04 同仓库更多 Skills
syllabus
Generates a curated supplementary reading list from any course syllabus using Consensus academic search. Grill-me intake (syllabus input format + course audience + year range) plus a grouping forcing-options checkpoint before any search runs — so the reading list matches the course's level and recency need. Parses the syllabus to extract topics and learning outcomes, searches Consensus for recent peer-reviewed papers per topic, and produces a professionally formatted .docx with clickable Consensus links, plain-language summaries calibrated to audience level, and Bloom-higher-order discussion questions tied to course learning goals. Use when the user uploads a syllabus, course outline, or curriculum document and wants supplementary readings (e.g., 'create a reading list from this syllabus', 'find recent papers for my course') — even casual mentions with a syllabus attached should trigger this skill.
research
Default entry point for any research request — a hybrid router that classifies the question deterministically and either delegates to a specialist research skill (pulse for trends/sentiment, grants for NIH funding, litreview for academic literature, syllabus for course reading, patent for prior-art + IP landscape, dossier for entity research) or runs its own plan-decompose-multi-source-search-synthesize-cite fallback workflow when no specialist matches. Always surfaces the routing decision so users can override. Use when the user makes any research request that doesn't obviously match a more-specific specialist skill (e.g., "research [topic]", "look into [topic]", "what do we know about [topic]", "investigate [topic]", "find me information on [topic]", "do some research on [topic]", "I need to understand [topic]"). Output is a markdown briefing (default) or .docx document (on request) with full citations and an audit log.
pulse
Multi-source recency research skill that takes the pulse of any topic across Reddit, Hacker News, the open web, and optionally X/Twitter within a configurable recent window (default 30 days). Forcing intake clarifies topic specificity, angle (trend/sentiment/problems/opportunities/comparison), time window, and platform scope before searching. Returns a synthesized briefing with citations, engagement metrics, and cross-platform pattern analysis. Use when the user requests multi-source recency intelligence on a topic (e.g., 'pulse on [topic]', 'what's happening with [topic]', 'what are people saying about [topic]', 'current conversation about [topic]', 'take the pulse of [topic]', 'trending: [topic]', 'find me info on [topic]'), and for competitor research, trend discovery, tool comparisons, and audience sentiment analysis.
patent
Patent prior-art and landscape intelligence skill — not generic patent help. Commits to one of five sub-use-cases via forcing intake (novelty search / freedom-to-operate / competitive landscape / acquisition diligence / litigation prior-art) before any search runs. Searches Google Patents, Espacenet, USPTO, and optionally Lens.org for citation-graph signals. Output is an editable Word document (.docx) with verdict, ranked closest art (claim-text extracted), CPC-class-aware landscape, family-resolved hits, geographic coverage, FTO flags where applicable, strategy recommendations, and full audit log. Use when the user asks for patent searching or analysis (e.g., 'prior art search for [invention]', 'freedom to operate analysis for [product]'). Produces search signal, not legal advice — always recommends consulting a patent attorney before filing or licensing decisions. Trademark, copyright, and trade-secret questions are out of scope.
notebooklm
Browser automation skill for controlling Google's NotebookLM. Use when the user wants anything done in NotebookLM (e.g., 'open NotebookLM', 'check my [name] notebook', 'ask my notebook about X', 'add [source] to NotebookLM', 'generate a Video Overview from my notebook', 'use NotebookLM Studio'). Handles reading and querying notebooks, adding sources (URLs, text, files, YouTube links, synthesized content), generating Studio outputs (Audio/Video Overviews, Mind Maps, Reports incl. Briefing Doc/Study Guide/FAQ, Flashcards, Quiz, slide decks, infographics — discover the exact set from the live Studio panel; the UI evolves fast), and creating new notebooks. Requires browser automation environment — fails gracefully when unavailable.
litreview
Academic literature orientation skill that searches papers via free keyless APIs (PubMed E-utilities + OpenAlex) by default — with the Consensus MCP as an optional enhancement lane when connected — builds a strategic search plan using PICO (default) or SPIDER / Decomposition / hybrid as fallbacks, and synthesizes findings into a formatted Word (.docx) research guide. Grill-me intake (research question specificity + framework hint + tentative depth) before the recon search; a second forcing checkpoint after Phase 2 confirms framework + sub-areas + depth before searches consume budget. Configurable depth (5/10/20 queries) controls coverage vs. speed. Output is a 'launching pad' — an orientation guide that lets a researcher dive in confidently, not a finished review. Use when the user starts literature-oriented research (e.g., 'litreview on [topic]', 'literature review on [topic]', 'I'm starting a literature review on X', 'I'm writing a paper on X', 'help me research X', 'I'm doing research on X', 'can you help me research X'). Do NOT use for single one-off paper searches wanting a quick list — that's a plain PubMed/OpenAlex (or Consensus) query.