AGENT SKILL

data-preparation

This skill prepares and preflights MapTR nuScenes and Argoverse2 data layouts, annotation contracts, and vector-map schemas before model execution.

License 许可
MIT
最近更新
2026-09-03
Stars
★ 351
Forks
27

01 前言 Front Matter

name
data-preparation
license
MIT
metadata
{ "disco-role": "operating" }
description
This skill prepares and preflights MapTR nuScenes and Argoverse2 data layouts, annotation contracts, and vector-map schemas before model execution.
disable-model-invocation
true

02 资源文件 4

03 SKILL.md 正文

MapTR data preparation

Use this route before training, evaluation, or visualization. It turns a raw nuScenes or Argoverse2 download into a checked layout plan and tells the next agent which generated files and vector schema MapTR expects. It does not download data, run a full converter, install packages, or modify a user data root.

Fast path

  1. Choose the dataset and make an explicit absolute data root.

  2. Run the bundled CPU-only layout checker. It checks directories and file names; it does not import MapTR, mmcv, nuscenes, av2, or torch.

    python <skill-root>/scripts/check_dataset_layout.py \
      --dataset nuscenes --root /data/maptr/nuscenes --canbus-root /data/maptr \
      --check-annotations
    python <skill-root>/scripts/check_dataset_layout.py \
      --dataset av2 --root /data/maptr/argoverse2/sensor \
      --check-annotations
    
  3. Resolve every ERROR before starting a converter. Use --json when a machine-readable report is needed. --self-test exercises valid and deliberately broken tiny fixtures without accessing external data.

  4. Select a converter only after the raw layout passes. Record the exact command, version, output root, and worker count; preserve converter logs.

  5. Re-run with --check-annotations after conversion. A passing layout check is not proof that images, calibration, transforms, or map geometry are numerically correct.

The checker returns exit code 0 only when all requested checks pass. It returns 1 for a missing required path, wrong placement, malformed map archive, or missing requested annotation file. It never creates a directory in the data root.

Installation and safety gates

The documented dependency target is Python 3.8, PyTorch 1.9.1+cu111, mmcv-full==1.4.0, mmdet==2.14.0, mmsegmentation==0.14.1, timm, mmdetection3d==0.17.2, shapely==1.8.5.post1, and the av2 API for AV2. The project also documents installing its mmdetection3d component and the Geometric Kernel Attention extension. Treat these versions as a compatibility plan, not as a verified installation on the current machine. A legacy CUDA compiler, matching mmcv-full, and a successful custom extension build are not assumed here.

Install prerequisites in an isolated environment and prove imports before conversion. Do not start with a large converter just to discover an import or ABI mismatch. See the workflow reference for the ordered probes and documented commands. Route model and extension choices to model-configuration, and route execution to training-evaluation.

Dataset choices and converter contracts

nuScenes

The documented raw root is data/nuscenes/, with maps/, samples/, sweeps/, and version metadata such as v1.0-trainval/ and v1.0-test/. The CAN bus expansion belongs beside the dataset root: with --root-path /data/maptr/nuscenes --canbus /data/maptr, the converter looks for /data/maptr/can_bus/, not /data/maptr/nuscenes/can_bus/.

Reference command for the full release:

python tools/create_data.py nuscenes \
  --root-path /data/maptr/nuscenes --out-dir /data/maptr/nuscenes \
  --extra-tag nuscenes --version v1.0 --canbus /data/maptr \
  --max-sweeps 10

tools/create_data.py dispatches nuscenes twice for a non-mini version: it passes v1.0-trainval, then v1.0-test, to the nuScenes converter. The trainval call writes nuscenes_infos_temporal_train.pkl and nuscenes_infos_temporal_val.pkl; the test call writes nuscenes_infos_temporal_test.pkl. A v1.0-mini request is handled once and uses the mini split. The converter also performs 2D annotation export. MapTR's map dataset consumes the temporal infos records and derives local vectors from the map expansion at runtime.

Argoverse2 Sensor

The documented root is data/argoverse2/sensor/ with exactly the split level train/, val/, and test/. Each log directory needs the sensor data used by the loader and one map archive matching map/log_map_archive_*.json. The converter is:

python tools/data_converter/av2_converter.py \
  --data-root /data/maptr/argoverse2/sensor --nproc 8

The CLI defaults to 64 workers; choose a bounded value for a first run. Its __main__ path processes train, val, and test, removes a fixed list of known failing logs, queries lidar timestamps, chooses the closest image for seven ring cameras, and discards samples with unsynchronized lidar/camera paths. It loads one JSON map archive per log and writes, at the sensor root, av2_map_infos_train.pkl, av2_map_infos_val.pkl, and av2_map_infos_test.pkl. The config normally consumes train and val files and uses load_interval=4 for validation.

Full converters are reference-only for this route: they require downloaded private data, third-party APIs, map parsing, multiprocessing, and writes to large output files. The checker is the safe preflight; do not replace it with a full conversion run on synthetic folders.

MapTR data contract

For nuScenes configs, set data_root to the nuScenes directory, ann_file to the matching temporal train/val pkl, and leave map_ann_file for the map evaluation-format JSON (for example, nuscenes_map_anns_val.json). map_ann_file is not a substitute for the raw map expansion or the temporal annotation pkl. For AV2, set data_root to the sensor directory and use av2_map_infos_{train,val}.pkl; the dataset loader expects each pkl to contain samples and id2map, sorted by sample timestamp. map_ann_file similarly names the map-format output used by validation/test formatting.

Both custom datasets expose three map classes in the standard MapTR configs: divider (label 0), ped_crossing (label 1), and boundary (label 2). VectorizedLocalMap obtains nuScenes divider lines, pedestrian-crossing polygons, and road/lane contour boundaries from the four supported map locations. VectorizedAV2LocalMap obtains divider polylines, pedestrian polygons, and drivable-area boundary polygons from the converter's id2map. Unexpected vector class names are errors; do not silently remap them.

The configured fixed-point representation is 20 points per ground-truth and predicted line. A nonempty LiDARInstanceLines.fixed_num_sampled_points has shape [N, 20, 2], in local x/y coordinates, clamped to the patch. Shifted representations may have shape [N, shifts, 20, 2]; closed polygons receive cyclic shifts, open lines receive forward/reverse variants, and padding uses -10000. Empty vector sets must remain representable; never fabricate a placeholder line. The patch is derived from pc_range: height is y_max-y_min, width is x_max-x_min. For the common nuScenes range [-15,-30,-2,15,30,2], the local patch is (60,30) and x/y limits are approximately [-15,15]/[-30,30]. AV2's common range reverses those planar extents.

Handoff and routing

Report: dataset and release, absolute roots, raw checks, converter command planned or run, generated pkl names, map-archive count, map classes, fixed point count, and any unresolved dependency or data-quality issue. Do not claim native conversion or model execution from a filesystem-only pass.

For configuration values and pc_range consistency use model-configuration. For train/test and runtime commands use training-evaluation. For map annotation rendering and benchmark plots use visualization-benchmarking. Use workflows, data formats, and troubleshooting for detailed recovery.

04 同仓库更多 Skills

VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-triton-opt

Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-solve

END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-skill

PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-pattern-fusion

Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-orchestrate

Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-feedback

Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).

★ 351