data-and-preprocessing
Prepare, inspect, and safely adapt MedicalDetectionToolkit array datasets, labels, preprocessing, patching, and batchgenerators pipelines for toy, LIDC, and PET-CT-style experiments.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- data-and-preprocessing
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Prepare, inspect, and safely adapt MedicalDetectionToolkit array datasets, labels, preprocessing, patching, and batchgenerators pipelines for toy, LIDC, and PET-CT-style experiments.
- disable-model-invocation
- true
02 资源文件 6
- 参考文档references/data-formats.md5.9 KB
- 参考文档references/data-loader-contracts.md6.0 KB
- 参考文档references/data-utilities.md5.2 KB
- 参考文档references/preprocessing-workflows.md5.1 KB
- 参考文档references/troubleshooting.md5.0 KB
- 脚本scripts/validate_preprocessed_case.py3.5 KB
03 SKILL.md 正文
Data and preprocessing
Use this sub-skill when a Researcher must turn a preprocessed medical-image case into the dictionaries and arrays expected by MedicalDetectionToolkit (MDT), or when adapting one of the example loaders without silently changing axes, label semantics, or patch coordinates. This is an operating reference, not a data acquisition recipe. The repository is legacy and explicitly no longer maintained; reproduce the contracts below and verify the installed environment before changing them.
Operating contract
Given an experiment config cf, a logger, a manifest/data directory, and
preprocessed image/segmentation arrays, produce one of these observable
contracts:
load_datasetreturns anOrderedDictkeyed bypid. Each value contains paths underdataandseg, thepid, and aclass_targetlist. LIDC and PET-CT additionally carryfg_slices; toy uses one array path for a packed two-plane[image, segmentation]file and does not carry foreground-slice metadata.- A training batch contains
data,seg,pid, andclass_target.datais batched channels-first andseghas a singleton segmentation-channel axis. The spatial axes are(x, y)for 2D or(x, y, z)for 3D at the point the network sees them; the exact loader transposes are documented in data-formats. - After
ConvertSegToBoundingBoxCoordinates, a batch also containsbb_targetandroi_labels. Patient iterators preserve patient-level targets aspatient_bb_targetandpatient_roi_labels, plusoriginal_img_shape; tiled batches additionally carrypatch_crop_coords.
Before loading a real case, validate a portable image/segmentation pair with
validate_preprocessed_case.py. The
validator accepts only .npy/.npz arrays and optional JSON metadata, has hard
size/voxel bounds, performs no repository imports, and never downloads or
modifies data. Its JSON metadata format is described in
troubleshooting.
Required workflow
- Select the evidence-backed family. Use the toy loader only for small 2D onboarding data; use LIDC for per-patient CT volumes with instance-valued ROIs and optional 2D/3D operation; use PET-CT for the two-channel CT/PET classification-style example with a binary foreground segmentation. Read preprocessing-workflows before changing normalization, resampling, or label creation.
- Freeze the manifest contract. The configured
input_df_nameis normallyinfo_df.pickle. Its rows must agree with the filenames derived frompid. Do not infer a different axis order from a filename. Checkclass_target,fg_slices(where present), and spacing/shape metadata before training. - Validate arrays and metadata. Use the bundled validator on a copied or synthetic case. Never pass arbitrary pickle files to a new utility. The source loaders read pandas pickle manifests, so treat them as trusted local inputs only and keep their paths outside portable skills.
- Construct the generator. Instantiate the appropriate
BatchGenerator, then usecreate_data_gen_pipeline. Training composes optional mirror andSpatialTransform; validation/testing usesCenterCropTransform; both then callConvertSegToBoundingBoxCoordinates. Exact flags and version-sensitive signatures are in data-loader-contracts. - Check labels after every geometric transform. Image and segmentation
must be transformed together. Use interpolation order 0 / background 0 for
segmentation. Confirm the resulting
bb_targetcoordinates fit the post-transform patch and that class labels still correspond to ROI ids. - Record reversible customization. Keep a small manifest schema note and a before/after shape example for every custom loader. If a change affects axes, label mode, patch size, or channel count, update config and tests together rather than patching a downstream model.
Label and geometry rules
- Background is label
0; positive integer ROI ids are foreground. In LIDC, each nonzero ROI id is an individual lesion andclass_targetis aligned to those ids after the preprocessing script'srixassignment. Do not relabel ROIs without updating the aligned class vector. - A pixelwise instance map has increasing ROI ids and needs
get_rois_from_seg_flag=False. A binary map has foreground1and does not identify individual lesions; useget_rois_from_seg_flag=Trueso the batchgenerators converter performs connected-component labeling. The PET-CT pipeline uses this binary mode. A box-only source must first be represented as a pseudo-mask if it is to go through the same spatial augmentation path. class_specific_seg_flagcontrols whether converted segmentation targets preserve class-specific channels/labels. It is not a substitute for making a binary mask into instance ids. Checkcf.head_classesandcf.num_seg_classeswhen changing it.- The loader may return no foreground in a sampled patch even when the patient
has lesions. Do not treat an empty patch as a corrupt case; distinguish it
from an invalid all-negative patient or an out-of-range
fg_slicesentry. - Coordinates are local to the current patch. Patient-level targets must be retained separately when tiling so predictions can be mapped/consolidated.
Dimensions, channels, and patching
- Toy is 2D only: its packed case stores image at index
0and segmentation at index1;BatchGeneratoradds a batch axis and a singleton data/seg channel axis. It uses320 x 320pre-crop and patch settings in the supplied config. - LIDC preprocessing writes
(z, y, x)image and ROI arrays. The loader transposes them to(y, x, z)before sampling; 2D mode selects a z slice, while 3D mode retains the volume. The supplied config uses300 x 300pre-crops /288 x 2882D patches, or156 x 156 x 96pre-crops /128 x 128 x 643D patches, and enables 2D-to-3D prediction merging. - PET-CT preprocessing writes a two-channel image
(c, z, y, x)and a binary(z, y, x)mask. The loader transposes the image to(c, y, x, z)and samples 2D slices or 3D volumes. The supplied config is 3D with280 x 280 x 48pre-crops and192 x 192 x 32patches, two channels, and 2D-to-3D merging enabled if 2D is selected. - If an input is smaller than the configured pre-crop or patient patch, the
loaders call
pad_nd_image; padding is centered and defaults to constant zero in the shown paths. If larger, training samples a pre-crop (with foreground-biased sampling) and patient inference creates a coordinate grid usingget_patch_crop_coords. A patch size of1in z means one patch per slice, not a three-dimensional slab. n_3D_contextadds neighboring slices as channels in 2D mode. It changescf.n_channelsand requires matching model input configuration; padding is applied at the volume boundary. Do not enable it without checking channel count and the context coordinate adjustment.
Safety boundaries
- The example raw data roots, mounted paths,
rsynccopies, and cloud/server branches are environment-specific. LIDC-IDRI, PET/CT/TNM, and the toy data directories are external datasets; the PET/CT paths and clinical labels may be private. Do not claim access, redistribute them, or run acquisition from this skill. The supplied examples are evidence of schema and behavior only. experiments/lidc_exp/pack_dataset.pyandutils/dataloader_utils.py::unpack_dataset/delete_npyare storage helpers, not safe validation steps. Packing/unpacking can create many files anddelete_npyis destructive. Treat pack, delete,rsync, multiprocessing preprocessing, and raw-data conversion as non-runnable unless a user explicitly reviews paths, backups, and a bounded dry run outside this skill.- Do not run native final cases, download data, or import this operating graph during construction. Use signatures/import checks and bounded synthetic validator fixtures only.
References and bundled script
- Data formats and array axes
- Data-loader and batchgenerators contracts
- Preprocessing workflows
- Data utilities: padding, tiling, balancing, packing boundaries
- Troubleshooting and safe customization
- Standalone preprocessed-case validator
All references link back here and to the other references so the graph remains
usable when a Researcher opens a leaf directly. Source evidence is the checked-
out README.md, the three experiment configs.py/data_loader.py pairs,
LIDC preprocessing.py/pack_dataset.py, PET-CT preprocessing.py, and
utils/dataloader_utils.py.
Verification gate
Accept a customized data path only when: (a) the three example loaders import
with the documented loader dependency/API variant; (b) the validator passes a
bounded synthetic valid case and rejects mismatched shapes, non-integral labels,
or oversized input;
(c) the manifest-to-file naming and class_target/ROI semantics are recorded;
and (d) no native dataset case or destructive helper was run. Unresolved
backend, private-data, axis, or label ambiguities stay explicit in the handoff.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).