data-preparation
Route for SSD Keras dataset parsing, validation, augmentation, and batch generation.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- data-preparation
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Route for SSD Keras dataset parsing, validation, augmentation, and batch generation.
- disable-model-invocation
- true
02 资源文件 2
- 参考文档references/troubleshooting.md2.1 KB
- 脚本scripts/smoke.py9.3 KB
03 SKILL.md 正文
Data Preparation
Use this route when the task is about getting object-detection data into a shape that the SSD workflows can use safely.
What this route covers
- parsing CSV annotations
- parsing Pascal VOC XML annotations
- parsing COCO-style JSON annotations
- loading and reloading HDF5 datasets
- validating boxes, labels, and image sizes
- generating batches with
DataGenerator - applying resize and augmentation chains
- converting predictions back to the original image frame with inverse transforms
What this route excludes
- building or compiling models
- loss configuration
- decoding predictions or evaluating mAP
- transfer learning / weight sampling
Send those tasks to training or inference-evaluation.
First things to check
- Open
references/data-formats.mdto confirm the expected annotation layout. - Open
references/compatibility.mdif imports are failing in the environment. - Open
references/model-architecture.mdif the data must feedSSDInputEncoder. - Run
scripts/smoke.pywhen you want a tiny synthetic parser / generator check.
Typical workflow
1. Choose the source format
- CSV is easiest for ad hoc detection datasets.
- VOC XML is the best fit for the Pascal VOC splits used in the notebooks.
- JSON is the right path for COCO-style datasets and COCO evaluation.
- HDF5 is useful when you want a reusable cached dataset for faster iteration.
2. Verify label conventions
- Keep background as class 0 on the model side.
- Make sure the label order is the one your parser and generator expect.
- Confirm that every transform sees the same
labels_formatmapping.
3. Parse and validate
- Use the parser that matches the dataset source.
- Drop or flag degenerate boxes early.
- Decide whether empty images should stay in the batch.
- If boxes disappear after cropping or scaling, loosen the validity filters before changing the labels themselves.
4. Generate batches
- Use
ResizeplusConvertTo3Channelsfor a simple deterministic smoke path. - Add augmentation chains only after the basic parser / generator flow is stable.
- For training, wire the generator to
SSDInputEncoderthrough the training route. - For evaluation, keep the inverse transforms so predictions can be mapped back to the original image coordinates.
5. Cache when it helps
- If the source annotations are expensive to parse repeatedly, build an HDF5 cache.
- Reuse the cache only when the source images and labels are still the same.
Useful source objects
DataGeneratorparse_csvparse_xmlparse_jsoncreate_hdf5_datasetgenerateBoxFilterImageValidatorBoundGeneratorResizeConvertTo3ChannelsSSDDataAugmentationDataAugmentationConstantInputSizeDataAugmentationVariableInputSizeDataAugmentationSatelliteapply_inverse_transforms
Quick decision guide
- Need a safe parser smoke? Use the bundled script and a tiny synthetic fixture.
- Need the full training path? Go to
trainingafter the data contract is fixed. - Need evaluation or visualisation with transformed images? Keep the inverse transforms and go to
inference-evaluation.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).