[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M79FYN90ACDVFX3AFMD7DYCF":3,"skill-same:01M79FYN90ACDVFX3AFMD7DYCF":128},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":34,"commitSha":35,"skillDirectory":36,"forksCount":14,"repositoryHtmlUrl":37,"repositoryHomepageUrl":6,"ownerHtmlUrl":38,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":39,"install":40,"resources":53},"01M79FYN90ACDVFX3AFMD7DYCF","tao-train-sparse4d",null,"Sparse4D for multi-camera temporal 3D object detection and tracking. Uses sparse queries with deformable attention across camera views and time for end-to-end 3D perception, with an instance bank for temporal tracking. Use when training, evaluating, exporting, quantizing, or running inference for a TAO Sparse4D model. Trigger phrases include \"train Sparse4D\", \"multi-camera 3D detection\", \"temporal 3D tracker\", \"sparse query 3D perception\".","Elogic360\u002Fsoftware-factory","Elogic360","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F156535519?v=4","01M74NH3F0RB13PWB7JZDE2V80","elogic360","01M74NH3FPC65082MDZVMFHDM3",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: tao-train-sparse4d\ndescription: Sparse4D for multi-camera temporal 3D object detection and tracking. Uses sparse queries with deformable\n  attention across camera views and time for end-to-end 3D perception, with an instance bank for temporal tracking. Use when\n  training, evaluating, exporting, quantizing, or running inference for a TAO Sparse4D model. Trigger phrases include\n  \"train Sparse4D\", \"multi-camera 3D detection\", \"temporal 3D tracker\", \"sparse query 3D perception\".\nlicense: Apache-2.0\ncompatibility: Requires docker + nvidia-container-toolkit.\nmetadata:\n  version: \"0.1.0\"\n  author: NVIDIA Corporation\nallowed-tools: Read Bash\ntags:\n- temporal\n- 3d\n- detection\n- tracking\n---\n\n# Sparse4D\n\nSparse4D for multi-camera temporal 3D object detection and tracking. Uses sparse queries with deformable attention across camera views and time for end-to-end 3D perception. Includes instance bank for temporal tracking.\n\nRequires pretrained ResNet-101 backbone. Set train.pretrained_model_path.\n\n## Dataclass Schemas\n\nGenerated TAO Core schemas are packaged in `schemas\u002F\u003Caction>.schema.json`, with `schemas\u002Fmanifest.json` listing available actions. Each generated schema also emits `references\u002Fspec_template_\u003Caction>.yaml` from the schema top-level `default` field. AutoML enablement is declared at the model layer in `references\u002Fskill_info.yaml` via `automl_enabled`. Runnable AutoML still requires `schemas\u002Ftrain.schema.json` and `references\u002Fspec_template_train.yaml` to exist and parse. Use the packaged train schema for `automl_default_parameters`, `automl_disabled_parameters`, defaults, min\u002Fmax bounds, enums, option weights, math conditions, dependencies, and popular parameters. Do not expect `~\u002Ftao-core` at runtime; maintainers regenerate schemas\u002Ftemplates before packaging the skill bank.\n\n## Train Action Policy\n\nThis model is AutoML-enabled at the model layer. Before handling any train-stage request, read `references\u002Fskill_info.yaml` and resolve the run override from either an explicit `automl_policy` value or the user's workflow request. Treat phrases like \"turn off AutoML\", \"disable AutoML\", \"no HPO\", or \"plain training\" as `automl_policy: off` for this run only; otherwise default to `auto`. When `automl_policy: auto`, `automl_enabled: true`, and both `schemas\u002Ftrain.schema.json` and `references\u002Fspec_template_train.yaml` are packaged, route the train action through `tao-skill-bank:tao-run-automl` by default with this model's `skill_dir`. Preserve workflow\u002Fapplication overrides for datasets, specs, output directories, GPU\u002Fplatform settings, parent checkpoints, and `automl_policy`. Use direct model training only when `automl_policy: off` or the packaged train schema\u002Ftemplate is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.\n\nNon-train actions such as `evaluate`, `inference`, `export`, and deploy flows stay in this model skill. The per-run `automl_policy` override does not change model metadata.\n\n## Training Requirements\n\n- **Dataset type:** sparse4d\n- **Formats:** ovpkl\n- **Monitoring metric:** val_mAP\n\n### Per-Action Dataset Requirements\n\n| Action | Spec Key | Source | Files | List? |\n|---|---|---|---|---|\n| dataset_convert | aicity.root | id |  | No |\n| evaluate | dataset.data_root | eval_dataset | (from convert job, spec: aicity.split) | No |\n| evaluate | model.head.instance_bank.anchor | train_datasets | \u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy | No |\n| evaluate | dataset.train_dataset.ann_file | train_datasets | (from convert job, spec: aicity.split) | No |\n| evaluate | dataset.val_dataset.ann_file | eval_dataset | (from convert job, spec: aicity.split) | No |\n| evaluate | dataset.test_dataset.ann_file | inference_dataset | (from convert job, spec: aicity.split) | No |\n| export | model.head.instance_bank.anchor | train_datasets | \u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy | No |\n| inference | dataset.data_root | inference_dataset | (from convert job, spec: aicity.split) | No |\n| inference | model.head.instance_bank.anchor | train_datasets | \u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy | No |\n| inference | dataset.train_dataset.ann_file | train_datasets | (from convert job, spec: aicity.split) | No |\n| inference | dataset.val_dataset.ann_file | eval_dataset | (from convert job, spec: aicity.split) | No |\n| inference | dataset.test_dataset.ann_file | inference_dataset | (from convert job, spec: aicity.split) | No |\n| quantize | dataset.data_root | train_datasets | (from convert job, spec: aicity.split) | No |\n| quantize | model.head.instance_bank.anchor | train_datasets | \u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy | No |\n| quantize | dataset.train_dataset.ann_file | train_datasets | (from convert job, spec: aicity.split) | No |\n| quantize | dataset.val_dataset.ann_file | eval_dataset | (from convert job, spec: aicity.split) | No |\n| quantize | dataset.test_dataset.ann_file | inference_dataset | (from convert job, spec: aicity.split) | No |\n| quantize | dataset.quant_calibration_dataset.images_dir | train_datasets |  | No |\n| train | dataset.data_root | train_datasets | (from convert job, spec: aicity.split) | No |\n| train | model.head.instance_bank.anchor | train_datasets | \u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy | No |\n| train | dataset.train_dataset.ann_file | train_datasets | (from convert job, spec: aicity.split) | No |\n| train | dataset.val_dataset.ann_file | eval_dataset | (from convert job, spec: aicity.split) | No |\n| train | dataset.test_dataset.ann_file | inference_dataset | (from convert job, spec: aicity.split) | No |\n\n### Typical Spec Overrides\n\nData source overrides are **mandatory for every action** — the agent MUST construct data source paths from the Per-Action Dataset Requirements table above and include them in `spec_overrides`.\n\n```python\nS3_TRAIN = \"s3:\u002F\u002Fbucket\u002Fdata\u002Ftrain\"\nS3_EVAL = \"s3:\u002F\u002Fbucket\u002Fdata\u002Feval\"\n```\n\n**train (mandatory data sources):**\n```python\n{\n    \"train.num_epochs\": 30,\n    \"train.checkpoint_interval\": 10,\n    \"train.validation_interval\": 10,\n    \"train.num_gpus\": 1,\n    \"dataset.sequences.split_num\": 90,\n    \"train_dataset.sequences_split_num\": 90,\n    \"dataset.data_root\": {\"spec\": f\"{S3_TRAIN}\u002Faicity.split)\"},\n    \"model.head.instance_bank.anchor\": f\"{S3_TRAIN}\u002F\u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy\",\n    \"dataset.train_dataset.ann_file\": {\"spec\": f\"{S3_TRAIN}\u002Faicity.split)\"},\n    \"dataset.val_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n    \"dataset.test_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n}\n```\n\n**evaluate (mandatory data sources):**\n```python\n{\n    \"dataset.data_root\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n    \"model.head.instance_bank.anchor\": f\"{S3_TRAIN}\u002F\u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy\",\n    \"dataset.train_dataset.ann_file\": {\"spec\": f\"{S3_TRAIN}\u002Faicity.split)\"},\n    \"dataset.val_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n    \"dataset.test_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n}\n```\n\n**export (mandatory data sources):**\n```python\n{\n    \"model.head.instance_bank.anchor\": f\"{S3_TRAIN}\u002F\u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy\",\n}\n```\n\n**inference (mandatory data sources):**\n```python\n{\n    \"dataset.data_root\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n    \"model.head.instance_bank.anchor\": f\"{S3_TRAIN}\u002F\u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy\",\n    \"dataset.train_dataset.ann_file\": {\"spec\": f\"{S3_TRAIN}\u002Faicity.split)\"},\n    \"dataset.val_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n    \"dataset.test_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n}\n```\n\n**quantize (mandatory data sources):**\n```python\n{\n    \"dataset.data_root\": {\"spec\": f\"{S3_TRAIN}\u002Faicity.split)\"},\n    \"model.head.instance_bank.anchor\": f\"{S3_TRAIN}\u002F\u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy\",\n    \"dataset.train_dataset.ann_file\": {\"spec\": f\"{S3_TRAIN}\u002Faicity.split)\"},\n    \"dataset.val_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n    \"dataset.test_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n    \"dataset.quant_calibration_dataset.images_dir\": f\"{S3_TRAIN}\",\n}\n```\n## Eval Dataset\n\nOptional. Val\u002Ftest splits configured via dataset ann_file paths.\n\n## Important Parameters\n\n- **model.backbone**: Backbone. Default resnet_101.\n- **model.neck.out_channels**: FPN output channels. Default 256. num_outs=4.\n- **model.input_shape**: Input image shape [W, H]. Default [1408, 512].\n- **model.head.num_output**: Number of detection output queries. Default 300.\n- **model.head.num_decoder**: Number of decoder layers. Default 6.\n- **model.head.temporal**: Enable temporal reasoning. Default True.\n- **model.head.instance_bank.num_anchor**: Instance bank anchors. Default 900.\n- **model.head.instance_bank.num_temp_instances**: Temporal instance count. Default 600.\n- **model.depth_branch.loss_weight**: Depth supervision loss weight. Default 0.2.\n- **dataset.batch_size**: Per-GPU batch size. Default 2.\n- **dataset.num_frames**: Sequence length. Default 200.\n- **dataset.classes**: Detection classes. Default [person, gr1_t2, agility_digit, nova_carter]. num_ids=70 for tracking.\n- **train.optim.lr**: Learning rate. Default 5e-5. img_backbone lr_mult=0.2.\n- **train.lr_scheduler**: Cosine scheduler with linear warmup (500 iters, ratio 0.333).\n- **train.grad_clip.max_norm**: Gradient clipping. Default 25.\n- **train.precision**: Options: bf16, fp16, fp32. Default bf16.\n- **evaluate.metrics**: Eval metrics. Default [\"detection\"]. Optional tracking evaluation.\n- **evaluate.tracking.enabled**: Enable tracking evaluation. tracking_threshold=0.2.\n\n## Multi-GPU \u002F Multi-Node\n\n**Launch method:** Lightning-managed (single `python` process, Lightning spawns workers).\n\n| Spec Key | Description | Default |\n|----------|-------------|---------|\n| `train.num_gpus` | Number of GPUs | 1 |\n| `train.gpu_ids` | GPU device indices | [0] |\n| `train.num_nodes` | Number of nodes | 1 |\n\n- Multi-GPU strategy: `ddp_find_unused_parameters_true` (no fsdp support)\n- `sync_batchnorm` is always enabled (True)\n- Iterations per epoch computed as: `num_frames * num_bev_groups \u002F (num_nodes * num_gpus * batch_size)`\n- **Scaling:** When increasing GPUs, effective batch size grows and iterations-per-epoch shrinks proportionally\n\n**Multi-node env vars** (set by orchestrator): `WORLD_SIZE`, `NODE_RANK`, `MASTER_ADDR`, `MASTER_PORT`, `NUM_GPU_PER_NODE`.\n\n## Hardware\n\nMinimum 2 GPU(s), recommended 8 GPU(s). 40GB+ (A100 recommended) VRAM per GPU. Multi-camera temporal model is memory intensive. bf16 required for practical training. Multi-GPU strongly recommended. Instance bank requires substantial memory for temporal reasoning.\n\n## Error Patterns\n\n**dataset_convert required**: Must run dataset_convert first to produce annotation pickles and anchor_init.npy.\n\n**Missing anchor file**: Set model.head.instance_bank.anchor to the anchor_init.npy path from dataset_convert results.\n\n**Temporal OOM**: Reduce dataset.num_frames or dataset.batch_size if running out of memory during temporal training.\n\n## Spec Param \u002F Parent Model Inference\n\nModel-specific inference mappings belong in this MD file, not in `config.json`. Generated runners should read this section and apply the mappings with SDK helpers before `create_job()`. This mirrors the old microservices `infer_params.py` flow.\n\nInference mappings from TAO Core `sparse4d.config.json`:\n\n| Action | Spec Field | Inference Function | Meaning |\n|---|---|---|---|\n| dataset_convert | `results_dir` | `output_dir` | current job results directory |\n| evaluate | `encryption_key` | `key` | encryption key |\n| evaluate | `evaluate.checkpoint` | `parent_model` | model file inferred from the parent job results folder |\n| evaluate | `results_dir` | `output_dir` | current job results directory |\n| export | `encryption_key` | `key` | encryption key |\n| export | `export.checkpoint` | `parent_model` | model file inferred from the parent job results folder |\n| export | `export.onnx_file` | `create_onnx_file` | output ONNX path |\n| export | `results_dir` | `output_dir` | current job results directory |\n| inference | `encryption_key` | `key` | encryption key |\n| inference | `inference.checkpoint` | `parent_model` | model file inferred from the parent job results folder |\n| inference | `results_dir` | `output_dir` | current job results directory |\n| quantize | `encryption_key` | `key` | encryption key |\n| quantize | `quantize.model_path` | `parent_model` | model file inferred from the parent job results folder |\n| quantize | `results_dir` | `output_dir` | current job results directory |\n| train | `encryption_key` | `key` | encryption key |\n| train | `results_dir` | `output_dir` | current job results directory |\n| train | `train.pretrained_model_path` | `ptm_if_no_resume_model` | PTM when no resume checkpoint exists |\n| train | `train.resume_training_checkpoint_path` | `resume_model` | model file inferred from the current job results folder |\n\nFor `parent_model` or `parent_model_folder`, pass the upstream train\u002Fexport\u002FAutoML child job id as `parent_job_id`. The SDK lists the parent result folder, filters checkpoint artifacts, and returns the selected model file or folder. Do not add these mappings back to `config.json` and do not patch generated runner scripts to guess checkpoint paths.\n","# Sparse4D\n\nSparse4D for multi-camera temporal 3D object detection and tracking. Uses sparse queries with deformable attention across camera views and time for end-to-end 3D perception. Includes instance bank for temporal tracking.\n\nRequires pretrained ResNet-101 backbone. Set train.pretrained_model_path.\n\n## Dataclass Schemas\n\nGenerated TAO Core schemas are packaged in `schemas\u002F\u003Caction>.schema.json`, with `schemas\u002Fmanifest.json` listing available actions. Each generated schema also emits `references\u002Fspec_template_\u003Caction>.yaml` from the schema top-level `default` field. AutoML enablement is declared at the model layer in `references\u002Fskill_info.yaml` via `automl_enabled`. Runnable AutoML still requires `schemas\u002Ftrain.schema.json` and `references\u002Fspec_template_train.yaml` to exist and parse. Use the packaged train schema for `automl_default_parameters`, `automl_disabled_parameters`, defaults, min\u002Fmax bounds, enums, option weights, math conditions, dependencies, and popular parameters. Do not expect `~\u002Ftao-core` at runtime; maintainers regenerate schemas\u002Ftemplates before packaging the skill bank.\n\n## Train Action Policy\n\nThis model is AutoML-enabled at the model layer. Before handling any train-stage request, read `references\u002Fskill_info.yaml` and resolve the run override from either an explicit `automl_policy` value or the user's workflow request. Treat phrases like \"turn off AutoML\", \"disable AutoML\", \"no HPO\", or \"plain training\" as `automl_policy: off` for this run only; otherwise default to `auto`. When `automl_policy: auto`, `automl_enabled: true`, and both `schemas\u002Ftrain.schema.json` and `references\u002Fspec_template_train.yaml` are packaged, route the train action through `tao-skill-bank:tao-run-automl` by default with this model's `skill_dir`. Preserve workflow\u002Fapplication overrides for datasets, specs, output directories, GPU\u002Fplatform settings, parent checkpoints, and `automl_policy`. Use direct model training only when `automl_policy: off` or the packaged train schema\u002Ftemplate is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.\n\nNon-train actions such as `evaluate`, `inference`, `export`, and deploy flows stay in this model skill. The per-run `automl_policy` override does not change model metadata.\n\n## Training Requirements\n\n- **Dataset type:** sparse4d\n- **Formats:** ovpkl\n- **Monitoring metric:** val_mAP\n\n### Per-Action Dataset Requirements\n\n| Action | Spec Key | Source | Files | List? |\n|---|---|---|---|---|\n| dataset_convert | aicity.root | id |  | No |\n| evaluate | dataset.data_root | eval_dataset | (from convert job, spec: aicity.split) | No |\n| evaluate | model.head.instance_bank.anchor | train_datasets | \u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy | No |\n| evaluate | dataset.train_dataset.ann_file | train_datasets | (from convert job, spec: aicity.split) | No |\n| evaluate | dataset.val_dataset.ann_file | eval_dataset | (from convert job, spec: aicity.split) | No |\n| evaluate | dataset.test_dataset.ann_file | inference_dataset | (from convert job, spec: aicity.split) | No |\n| export | model.head.instance_bank.anchor | train_datasets | \u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy | No |\n| inference | dataset.data_root | inference_dataset | (from convert job, spec: aicity.split) | No |\n| inference | model.head.instance_bank.anchor | train_datasets | \u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy | No |\n| inference | dataset.train_dataset.ann_file | train_datasets | (from convert job, spec: aicity.split) | No |\n| inference | dataset.val_dataset.ann_file | eval_dataset | (from convert job, spec: aicity.split) | No |\n| inference | dataset.test_dataset.ann_file | inference_dataset | (from convert job, spec: aicity.split) | No |\n| quantize | dataset.data_root | train_datasets | (from convert job, spec: aicity.split) | No |\n| quantize | model.head.instance_bank.anchor | train_datasets | \u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy | No |\n| quantize | dataset.train_dataset.ann_file | train_datasets | (from convert job, spec: aicity.split) | No |\n| quantize | dataset.val_dataset.ann_file | eval_dataset | (from convert job, spec: aicity.split) | No |\n| quantize | dataset.test_dataset.ann_file | inference_dataset | (from convert job, spec: aicity.split) | No |\n| quantize | dataset.quant_calibration_dataset.images_dir | train_datasets |  | No |\n| train | dataset.data_root | train_datasets | (from convert job, spec: aicity.split) | No |\n| train | model.head.instance_bank.anchor | train_datasets | \u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy | No |\n| train | dataset.train_dataset.ann_file | train_datasets | (from convert job, spec: aicity.split) | No |\n| train | dataset.val_dataset.ann_file | eval_dataset | (from convert job, spec: aicity.split) | No |\n| train | dataset.test_dataset.ann_file | inference_dataset | (from convert job, spec: aicity.split) | No |\n\n### Typical Spec Overrides\n\nData source overrides are **mandatory for every action** — the agent MUST construct data source paths from the Per-Action Dataset Requirements table above and include them in `spec_overrides`.\n\n```python\nS3_TRAIN = \"s3:\u002F\u002Fbucket\u002Fdata\u002Ftrain\"\nS3_EVAL = \"s3:\u002F\u002Fbucket\u002Fdata\u002Feval\"\n```\n\n**train (mandatory data sources):**\n```python\n{\n    \"train.num_epochs\": 30,\n    \"train.checkpoint_interval\": 10,\n    \"train.validation_interval\": 10,\n    \"train.num_gpus\": 1,\n    \"dataset.sequences.split_num\": 90,\n    \"train_dataset.sequences_split_num\": 90,\n    \"dataset.data_root\": {\"spec\": f\"{S3_TRAIN}\u002Faicity.split)\"},\n    \"model.head.instance_bank.anchor\": f\"{S3_TRAIN}\u002F\u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy\",\n    \"dataset.train_dataset.ann_file\": {\"spec\": f\"{S3_TRAIN}\u002Faicity.split)\"},\n    \"dataset.val_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n    \"dataset.test_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n}\n```\n\n**evaluate (mandatory data sources):**\n```python\n{\n    \"dataset.data_root\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n    \"model.head.instance_bank.anchor\": f\"{S3_TRAIN}\u002F\u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy\",\n    \"dataset.train_dataset.ann_file\": {\"spec\": f\"{S3_TRAIN}\u002Faicity.split)\"},\n    \"dataset.val_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n    \"dataset.test_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n}\n```\n\n**export (mandatory data sources):**\n```python\n{\n    \"model.head.instance_bank.anchor\": f\"{S3_TRAIN}\u002F\u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy\",\n}\n```\n\n**inference (mandatory data sources):**\n```python\n{\n    \"dataset.data_root\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n    \"model.head.instance_bank.anchor\": f\"{S3_TRAIN}\u002F\u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy\",\n    \"dataset.train_dataset.ann_file\": {\"spec\": f\"{S3_TRAIN}\u002Faicity.split)\"},\n    \"dataset.val_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n    \"dataset.test_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n}\n```\n\n**quantize (mandatory data sources):**\n```python\n{\n    \"dataset.data_root\": {\"spec\": f\"{S3_TRAIN}\u002Faicity.split)\"},\n    \"model.head.instance_bank.anchor\": f\"{S3_TRAIN}\u002F\u002Fresults\u002F{dataset_convert_job_id}\u002Fanchor_init.npy\",\n    \"dataset.train_dataset.ann_file\": {\"spec\": f\"{S3_TRAIN}\u002Faicity.split)\"},\n    \"dataset.val_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n    \"dataset.test_dataset.ann_file\": {\"spec\": f\"{S3_EVAL}\u002Faicity.split)\"},\n    \"dataset.quant_calibration_dataset.images_dir\": f\"{S3_TRAIN}\",\n}\n```\n## Eval Dataset\n\nOptional. Val\u002Ftest splits configured via dataset ann_file paths.\n\n## Important Parameters\n\n- **model.backbone**: Backbone. Default resnet_101.\n- **model.neck.out_channels**: FPN output channels. Default 256. num_outs=4.\n- **model.input_shape**: Input image shape [W, H]. Default [1408, 512].\n- **model.head.num_output**: Number of detection output queries. Default 300.\n- **model.head.num_decoder**: Number of decoder layers. Default 6.\n- **model.head.temporal**: Enable temporal reasoning. Default True.\n- **model.head.instance_bank.num_anchor**: Instance bank anchors. Default 900.\n- **model.head.instance_bank.num_temp_instances**: Temporal instance count. Default 600.\n- **model.depth_branch.loss_weight**: Depth supervision loss weight. Default 0.2.\n- **dataset.batch_size**: Per-GPU batch size. Default 2.\n- **dataset.num_frames**: Sequence length. Default 200.\n- **dataset.classes**: Detection classes. Default [person, gr1_t2, agility_digit, nova_carter]. num_ids=70 for tracking.\n- **train.optim.lr**: Learning rate. Default 5e-5. img_backbone lr_mult=0.2.\n- **train.lr_scheduler**: Cosine scheduler with linear warmup (500 iters, ratio 0.333).\n- **train.grad_clip.max_norm**: Gradient clipping. Default 25.\n- **train.precision**: Options: bf16, fp16, fp32. Default bf16.\n- **evaluate.metrics**: Eval metrics. Default [\"detection\"]. Optional tracking evaluation.\n- **evaluate.tracking.enabled**: Enable tracking evaluation. tracking_threshold=0.2.\n\n## Multi-GPU \u002F Multi-Node\n\n**Launch method:** Lightning-managed (single `python` process, Lightning spawns workers).\n\n| Spec Key | Description | Default |\n|----------|-------------|---------|\n| `train.num_gpus` | Number of GPUs | 1 |\n| `train.gpu_ids` | GPU device indices | [0] |\n| `train.num_nodes` | Number of nodes | 1 |\n\n- Multi-GPU strategy: `ddp_find_unused_parameters_true` (no fsdp support)\n- `sync_batchnorm` is always enabled (True)\n- Iterations per epoch computed as: `num_frames * num_bev_groups \u002F (num_nodes * num_gpus * batch_size)`\n- **Scaling:** When increasing GPUs, effective batch size grows and iterations-per-epoch shrinks proportionally\n\n**Multi-node env vars** (set by orchestrator): `WORLD_SIZE`, `NODE_RANK`, `MASTER_ADDR`, `MASTER_PORT`, `NUM_GPU_PER_NODE`.\n\n## Hardware\n\nMinimum 2 GPU(s), recommended 8 GPU(s). 40GB+ (A100 recommended) VRAM per GPU. Multi-camera temporal model is memory intensive. bf16 required for practical training. Multi-GPU strongly recommended. Instance bank requires substantial memory for temporal reasoning.\n\n## Error Patterns\n\n**dataset_convert required**: Must run dataset_convert first to produce annotation pickles and anchor_init.npy.\n\n**Missing anchor file**: Set model.head.instance_bank.anchor to the anchor_init.npy path from dataset_convert results.\n\n**Temporal OOM**: Reduce dataset.num_frames or dataset.batch_size if running out of memory during temporal training.\n\n## Spec Param \u002F Parent Model Inference\n\nModel-specific inference mappings belong in this MD file, not in `config.json`. Generated runners should read this section and apply the mappings with SDK helpers before `create_job()`. This mirrors the old microservices `infer_params.py` flow.\n\nInference mappings from TAO Core `sparse4d.config.json`:\n\n| Action | Spec Field | Inference Function | Meaning |\n|---|---|---|---|\n| dataset_convert | `results_dir` | `output_dir` | current job results directory |\n| evaluate | `encryption_key` | `key` | encryption key |\n| evaluate | `evaluate.checkpoint` | `parent_model` | model file inferred from the parent job results folder |\n| evaluate | `results_dir` | `output_dir` | current job results directory |\n| export | `encryption_key` | `key` | encryption key |\n| export | `export.checkpoint` | `parent_model` | model file inferred from the parent job results folder |\n| export | `export.onnx_file` | `create_onnx_file` | output ONNX path |\n| export | `results_dir` | `output_dir` | current job results directory |\n| inference | `encryption_key` | `key` | encryption key |\n| inference | `inference.checkpoint` | `parent_model` | model file inferred from the parent job results folder |\n| inference | `results_dir` | `output_dir` | current job results directory |\n| quantize | `encryption_key` | `key` | encryption key |\n| quantize | `quantize.model_path` | `parent_model` | model file inferred from the parent job results folder |\n| quantize | `results_dir` | `output_dir` | current job results directory |\n| train | `encryption_key` | `key` | encryption key |\n| train | `results_dir` | `output_dir` | current job results directory |\n| train | `train.pretrained_model_path` | `ptm_if_no_resume_model` | PTM when no resume checkpoint exists |\n| train | `train.resume_training_checkpoint_path` | `resume_model` | model file inferred from the current job results folder |\n\nFor `parent_model` or `parent_model_folder`, pass the upstream train\u002Fexport\u002FAutoML child job id as `parent_job_id`. The SDK lists the parent result folder, filters checkpoint artifacts, and returns the selected model file or folder. Do not add these mappings back to `config.json` and do not patch generated runner scripts to guess checkpoint paths.\n",{"name":5,"tags":24,"license":15,"metadata":29,"description":7,"allowed-tools":32,"compatibility":33},[25,26,27,28],"temporal","3d","detection","tracking",{"author":30,"version":31},"NVIDIA Corporation","0.1.0","Read Bash","Requires docker + nvidia-container-toolkit.","main","c475fc64e94e803b50d14a791dfee8853fbb2c5c","skills\u002Fnvidia\u002Ftao-train-sparse4d","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory","https:\u002F\u002Fgithub.com\u002FElogic360","2026-11-08 22:21:22",{"source":8,"skillName":5,"commands":41,"prompt":51,"zipAvailable":52},[42,45,48],{"tool":43,"command":44},"npx","npx skills add Elogic360\u002Fsoftware-factory --skill tao-train-sparse4d",{"tool":46,"command":47},"bunx","bunx skills add Elogic360\u002Fsoftware-factory --skill tao-train-sparse4d",{"tool":49,"command":50},"pnpm","pnpm dlx skills add Elogic360\u002Fsoftware-factory --skill tao-train-sparse4d","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002FSKILL.md",true,[54,60,64,68,72,76,80,84,87,90,94,98,102,106,110,114,118,122],{"path":55,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":58,"sourceHtmlUrl":59},"BENCHMARK.md","REFERENCE","参考文档",4405,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002FBENCHMARK.md",{"path":61,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":62,"sourceHtmlUrl":63},"evals\u002Fevals.json",750,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Fevals\u002Fevals.json",{"path":65,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Fskill_info.yaml",1689,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Freferences\u002Fskill_info.yaml",{"path":69,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":70,"sourceHtmlUrl":71},"references\u002Fspec_template_dataset_convert.yaml",6930,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Freferences\u002Fspec_template_dataset_convert.yaml",{"path":73,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Fspec_template_evaluate.yaml",7130,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Freferences\u002Fspec_template_evaluate.yaml",{"path":77,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Fspec_template_export.yaml",7146,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Freferences\u002Fspec_template_export.yaml",{"path":81,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":82,"sourceHtmlUrl":83},"references\u002Fspec_template_inference.yaml",7163,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Freferences\u002Fspec_template_inference.yaml",{"path":85,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":70,"sourceHtmlUrl":86},"references\u002Fspec_template_quantize.yaml","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Freferences\u002Fspec_template_quantize.yaml",{"path":88,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":70,"sourceHtmlUrl":89},"references\u002Fspec_template_train.yaml","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Freferences\u002Fspec_template_train.yaml",{"path":91,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":92,"sourceHtmlUrl":93},"schemas\u002Fdataset_convert.schema.json",109884,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Fschemas\u002Fdataset_convert.schema.json",{"path":95,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":96,"sourceHtmlUrl":97},"schemas\u002Fevaluate.schema.json",113837,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Fschemas\u002Fevaluate.schema.json",{"path":99,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":100,"sourceHtmlUrl":101},"schemas\u002Fexport.schema.json",113416,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Fschemas\u002Fexport.schema.json",{"path":103,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":104,"sourceHtmlUrl":105},"schemas\u002Finference.schema.json",113971,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Fschemas\u002Finference.schema.json",{"path":107,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":108,"sourceHtmlUrl":109},"schemas\u002Fmanifest.json",18676,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Fschemas\u002Fmanifest.json",{"path":111,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":112,"sourceHtmlUrl":113},"schemas\u002Fquantize.schema.json",109870,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Fschemas\u002Fquantize.schema.json",{"path":115,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":116,"sourceHtmlUrl":117},"schemas\u002Ftrain.schema.json",109864,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Fschemas\u002Ftrain.schema.json",{"path":119,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":120,"sourceHtmlUrl":121},"skill-card.md",4219,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Fskill-card.md",{"path":123,"resourceType":124,"resourceTypeDesc":125,"sizeBytes":126,"sourceHtmlUrl":127},"skill.oms.sig","OTHER","其他",8045,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-train-sparse4d\u002Fskill.oms.sig",[129,136,143,150,157,164],{"skillNo":130,"name":131,"nameZh":6,"description":132,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":133,"categories":134,"keywords":135,"publishedAt":19},"01M79G206MT3J6S0GY3C45P2KV","vss-summarize-video","Use to summarize a recorded video via the LVS summarization microservice (HITL-gated) with a VLM fallback. Not for report generation or live RTSP captioning.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-summarize-video\u002FSKILL.md",[],[],{"skillNo":137,"name":138,"nameZh":6,"description":139,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":140,"categories":141,"keywords":142,"publishedAt":19},"01M79G1WN63PGRE8CKD47WEBQ5","vss-setup-video-analytics-api","Use to deploy the vss-video-analytics-api REST service standalone (config-source, data-log bind, Elasticsearch, optional Kafka). Not for full warehouse deploy.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-setup-video-analytics-api\u002FSKILL.md",[],[],{"skillNo":144,"name":145,"nameZh":6,"description":146,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":147,"categories":148,"keywords":149,"publishedAt":19},"01M79G1S9F7VY3EF6H53PRP5AN","vss-setup-behavior-analytics","Use to deploy the vss-behavior-analytics service standalone (entrypoint, config-source, optional calibration). Not for the full warehouse deploy.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-setup-behavior-analytics\u002FSKILL.md",[],[],{"skillNo":151,"name":152,"nameZh":6,"description":153,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":154,"categories":155,"keywords":156,"publishedAt":19},"01M79G1N1S1Y27PFZPYMNK1MTQ","vss-search-archive","Use this skill to run top-level VSS fusion search on archived video, or to ingest video files \u002F RTSP streams for search. Do NOT use for ad-hoc visual Q&A (use vss-ask-video), live captioning (use vss-deploy-dense-captioning), or video summarization and reports (use vss-summarize-video).","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-search-archive\u002FSKILL.md",[],[],{"skillNo":158,"name":159,"nameZh":6,"description":160,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":161,"categories":162,"keywords":163,"publishedAt":19},"01M79G1H6WN0Q6C0VCAXCZX6B7","vss-query-analytics","Use this skill when reading video-analytics metrics, incidents, alerts, and sensor data via the VA-MCP server (port 9901). Not for live VLM or incident-range narrative reports.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-query-analytics\u002FSKILL.md",[],[],{"skillNo":165,"name":166,"nameZh":6,"description":167,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":168,"categories":169,"keywords":170,"publishedAt":19},"01M79G1DYPNXQS7QHZ0AAF5FX5","vss-manage-video-io-storage","Use to call the VIOS REST API (sensor list, timelines, clip extraction, snapshots, add\u002Fdelete sensors and streams). Not for VLM inference or search.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-manage-video-io-storage\u002FSKILL.md",[],[]]