data-quality-fixes
Use AutoViz data-quality reports and FixDQ to inspect and repair tabular data issues.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- data-quality-fixes
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Use AutoViz data-quality reports and FixDQ to inspect and repair tabular data issues.
- disable-model-invocation
- true
02 资源文件 3
- 参考文档references/troubleshooting.md1.5 KB
- 参考文档references/workflows.md1.5 KB
- 脚本scripts/fixdq_smoke.py906 B
03 SKILL.md 正文
Data Quality Fixes
Use this sub-skill when the user asks about FixDQ, data_cleaning_suggestions, duplicate rows, mixed data types, infinities, rare categories, leakage, skew, outliers, or data-cleaning advice that appears during an AutoViz run.
Use this when
- The prompt names
FixDQ,Fix_DQ,data_cleaning_suggestions, ordq_report. - AutoViz prints data-quality warnings before plotting.
- The user wants a transformer-like cleanup step they can reuse on train/test data.
- The user is debugging
pandas_dq, pandas compatibility, or missingIPython.display. - The user wants to know whether a noisy dataset should be cleaned before plotting.
Core flow
- Start with a pandas DataFrame.
- Use
data_cleaning_suggestions(df, target=target)for a report-style inspection. - Use
FixDQ()when the user wants a fit/transform style cleaning object. - Keep target handling explicit: pass a target column name, a target list,
"", orNoneaccording to the user's problem. - After cleaning, hand the resulting DataFrame back to the EDA sub-skill for visualization.
- When the user only wants diagnosis, stop at the report and explain the result in plain language.
Read these references
references/workflows.md: report and transformer recipes.references/troubleshooting.md: pandas,IPython, andpandas_dqcompatibility notes.../../references/install-and-compatibility.md: package-version guidance.../../references/api-reference.md: signatures forFixDQanddata_cleaning_suggestions.../../references/troubleshooting.md: cross-cutting environment issues that can break the report path.
Use these scripts
- Run
scripts/fixdq_smoke.pyto verify thatFixDQanddata_cleaning_suggestionscan be imported and exercised on a tiny DataFrame. - If the failure is really plot rendering or
chart_format, switch to the EDA sub-skill and run its smoke script. - If the failure looks like a package install issue, run
../../scripts/inspect_install.pyfirst.
Important compatibility facts
data_cleaning_suggestionsdelegates topandas_dq.dq_report.- This repository version works with pandas 2.x; pandas 3.x removed
DataFrame.applymap, which can breakpandas_dq. pandas_dqimportsIPython.display, so a non-notebook environment may still needIPythoninstalled.FixDQ.__init__acceptsquantile,cat_fill_value,num_fill_value,rare_threshold, andcorrelation_threshold.- The report path is still useful even when the user never wants plots.
Issues the report can surface
- duplicate rows or duplicate columns
- zero-variance features
- rare categories
- high-cardinality features
- infinite values
- mixed Python types in a single column
- skewed distributions
- highly correlated features or leakage
- imbalanced classes
- target-related issues when a target column is supplied
Cross-routing
- If the user asks for saved charts or automated EDA plots after cleaning, route to
../eda-visualization/SKILL.md. - If the data-quality issue is specific to long text columns or wordcloud behavior, route to
../text-wordclouds/SKILL.md. - Keep dependency/environment fixes in references; do not leak private inspection-environment paths.
- If the user wants to apply the same cleanup to train and test data, emphasize
FixDQover the report-only helper.
Troubleshooting reminders
- If the report fails at import time, check
IPythonfirst, thenpandasversion, then XGBoost/setuptools interactions. - If a tiny sample gives odd warnings, explain that the dataset may be too small to classify reliably.
- If the target column is missing or misspelled, verify the exact column name before retrying.
- If the result object is a
Styleror other display wrapper, describe that the report succeeded even if the object is not a plain DataFrame.
Escalation
If the user wants to continue from the report into plotting, route back to the EDA sub-skill with the cleaned DataFrame. If the user wants text-column advice rather than numeric or categorical cleanup, route to the text sub-skill instead of stretching this one.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).