AGENT SKILL

data-preparation

Route for SSD Keras dataset parsing, validation, augmentation, and batch generation.

License 许可
Apache-2.0
最近更新
2026-09-03
Stars
★ 351
Forks
27

01 前言 Front Matter

name
data-preparation
license
Apache 2.0
metadata
{ "disco-role": "operating" }
description
Route for SSD Keras dataset parsing, validation, augmentation, and batch generation.
disable-model-invocation
true

02 资源文件 2

03 SKILL.md 正文

Data Preparation

Use this route when the task is about getting object-detection data into a shape that the SSD workflows can use safely.

What this route covers

  • parsing CSV annotations
  • parsing Pascal VOC XML annotations
  • parsing COCO-style JSON annotations
  • loading and reloading HDF5 datasets
  • validating boxes, labels, and image sizes
  • generating batches with DataGenerator
  • applying resize and augmentation chains
  • converting predictions back to the original image frame with inverse transforms

What this route excludes

  • building or compiling models
  • loss configuration
  • decoding predictions or evaluating mAP
  • transfer learning / weight sampling

Send those tasks to training or inference-evaluation.

First things to check

  1. Open references/data-formats.md to confirm the expected annotation layout.
  2. Open references/compatibility.md if imports are failing in the environment.
  3. Open references/model-architecture.md if the data must feed SSDInputEncoder.
  4. Run scripts/smoke.py when you want a tiny synthetic parser / generator check.

Typical workflow

1. Choose the source format

  • CSV is easiest for ad hoc detection datasets.
  • VOC XML is the best fit for the Pascal VOC splits used in the notebooks.
  • JSON is the right path for COCO-style datasets and COCO evaluation.
  • HDF5 is useful when you want a reusable cached dataset for faster iteration.

2. Verify label conventions

  • Keep background as class 0 on the model side.
  • Make sure the label order is the one your parser and generator expect.
  • Confirm that every transform sees the same labels_format mapping.

3. Parse and validate

  • Use the parser that matches the dataset source.
  • Drop or flag degenerate boxes early.
  • Decide whether empty images should stay in the batch.
  • If boxes disappear after cropping or scaling, loosen the validity filters before changing the labels themselves.

4. Generate batches

  • Use Resize plus ConvertTo3Channels for a simple deterministic smoke path.
  • Add augmentation chains only after the basic parser / generator flow is stable.
  • For training, wire the generator to SSDInputEncoder through the training route.
  • For evaluation, keep the inverse transforms so predictions can be mapped back to the original image coordinates.

5. Cache when it helps

  • If the source annotations are expensive to parse repeatedly, build an HDF5 cache.
  • Reuse the cache only when the source images and labels are still the same.

Useful source objects

  • DataGenerator
  • parse_csv
  • parse_xml
  • parse_json
  • create_hdf5_dataset
  • generate
  • BoxFilter
  • ImageValidator
  • BoundGenerator
  • Resize
  • ConvertTo3Channels
  • SSDDataAugmentation
  • DataAugmentationConstantInputSize
  • DataAugmentationVariableInputSize
  • DataAugmentationSatellite
  • apply_inverse_transforms

Quick decision guide

  • Need a safe parser smoke? Use the bundled script and a tiny synthetic fixture.
  • Need the full training path? Go to training after the data contract is fixed.
  • Need evaluation or visualisation with transformed images? Keep the inverse transforms and go to inference-evaluation.

04 同仓库更多 Skills

VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-triton-opt

Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-solve

END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-skill

PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-pattern-fusion

Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-orchestrate

Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-feedback

Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).

★ 351