AGENT SKILL

data-quality-fixes

Use AutoViz data-quality reports and FixDQ to inspect and repair tabular data issues.

License 许可
Apache-2.0
最近更新
2026-09-03
Stars
★ 351
Forks
27

01 前言 Front Matter

name
data-quality-fixes
license
Apache 2.0
metadata
{ "disco-role": "operating" }
description
Use AutoViz data-quality reports and FixDQ to inspect and repair tabular data issues.
disable-model-invocation
true

02 资源文件 3

03 SKILL.md 正文

Data Quality Fixes

Use this sub-skill when the user asks about FixDQ, data_cleaning_suggestions, duplicate rows, mixed data types, infinities, rare categories, leakage, skew, outliers, or data-cleaning advice that appears during an AutoViz run.

Use this when

  • The prompt names FixDQ, Fix_DQ, data_cleaning_suggestions, or dq_report.
  • AutoViz prints data-quality warnings before plotting.
  • The user wants a transformer-like cleanup step they can reuse on train/test data.
  • The user is debugging pandas_dq, pandas compatibility, or missing IPython.display.
  • The user wants to know whether a noisy dataset should be cleaned before plotting.

Core flow

  1. Start with a pandas DataFrame.
  2. Use data_cleaning_suggestions(df, target=target) for a report-style inspection.
  3. Use FixDQ() when the user wants a fit/transform style cleaning object.
  4. Keep target handling explicit: pass a target column name, a target list, "", or None according to the user's problem.
  5. After cleaning, hand the resulting DataFrame back to the EDA sub-skill for visualization.
  6. When the user only wants diagnosis, stop at the report and explain the result in plain language.

Read these references

Use these scripts

  • Run scripts/fixdq_smoke.py to verify that FixDQ and data_cleaning_suggestions can be imported and exercised on a tiny DataFrame.
  • If the failure is really plot rendering or chart_format, switch to the EDA sub-skill and run its smoke script.
  • If the failure looks like a package install issue, run ../../scripts/inspect_install.py first.

Important compatibility facts

  • data_cleaning_suggestions delegates to pandas_dq.dq_report.
  • This repository version works with pandas 2.x; pandas 3.x removed DataFrame.applymap, which can break pandas_dq.
  • pandas_dq imports IPython.display, so a non-notebook environment may still need IPython installed.
  • FixDQ.__init__ accepts quantile, cat_fill_value, num_fill_value, rare_threshold, and correlation_threshold.
  • The report path is still useful even when the user never wants plots.

Issues the report can surface

  • duplicate rows or duplicate columns
  • zero-variance features
  • rare categories
  • high-cardinality features
  • infinite values
  • mixed Python types in a single column
  • skewed distributions
  • highly correlated features or leakage
  • imbalanced classes
  • target-related issues when a target column is supplied

Cross-routing

  • If the user asks for saved charts or automated EDA plots after cleaning, route to ../eda-visualization/SKILL.md.
  • If the data-quality issue is specific to long text columns or wordcloud behavior, route to ../text-wordclouds/SKILL.md.
  • Keep dependency/environment fixes in references; do not leak private inspection-environment paths.
  • If the user wants to apply the same cleanup to train and test data, emphasize FixDQ over the report-only helper.

Troubleshooting reminders

  • If the report fails at import time, check IPython first, then pandas version, then XGBoost/setuptools interactions.
  • If a tiny sample gives odd warnings, explain that the dataset may be too small to classify reliably.
  • If the target column is missing or misspelled, verify the exact column name before retrying.
  • If the result object is a Styler or other display wrapper, describe that the report succeeded even if the object is not a plain DataFrame.

Escalation

If the user wants to continue from the report into plotting, route back to the EDA sub-skill with the cleaned DataFrame. If the user wants text-column advice rather than numeric or categorical cleanup, route to the text sub-skill instead of stretching this one.

04 同仓库更多 Skills

VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-triton-opt

Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-solve

END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-skill

PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-pattern-fusion

Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-orchestrate

Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-feedback

Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).

★ 351