data-pipelines
Use and debug Scenic dataset registry, dataset configs, input pipelines, BigTransfer preprocessing, FlexIO, COCO utilities, and data-format assumptions without downloading data by default.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-08
- Stars
- ★ 11
- Forks
- 3
01 前言 Front Matter
- name
- data-pipelines
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Use and debug Scenic dataset registry, dataset configs, input pipelines, BigTransfer preprocessing, FlexIO, COCO utilities, and data-format assumptions without downloading data by default.
- disable-model-invocation
- true
02 资源文件 3
- 参考文档references/data-formats.md9.7 KB
- 参考文档references/data-pipelines.md14.0 KB
- 参考文档references/troubleshooting.md8.8 KB
03 SKILL.md 正文
Scenic data pipelines
Use this sub-skill when the task is about Scenic dataset registration, dataset_name, dataset_configs, train_utils.get_dataset, TFDS-backed inputs, tf.data service behavior, BigTransfer (bit) preprocessing, FlexIO, COCO label utilities, or TFRecord/COCO dataset preflights.
Read/run map
- Read references/data-pipelines.md for the dataset registry,
@add_dataset, lazy imports,train_utils.get_dataset, TFDS loading, sharding, padding, prefetching, BigTransfer, and FlexIO pipeline construction. - Read references/data-formats.md for common batch keys,
meta_data, config fields, TFDS/COCO/FlexIO/TFRecord expectations, and when actual external data is required. - Read references/troubleshooting.md for unknown datasets, missing TFDS data, optional dependency failures, TensorFlow GPU-memory behavior, BigTransfer/TensorFlow Addons issues, FlexIO/custom preprocessing failures, and
tf.dataservice problems. - Run scripts/check_dataset_registry.py when you need a safe registry check. It lists Scenic's lazy dataset names and can optionally import a dataset module or run a registry lookup without constructing a dataset or downloading data.
Operating rules
- Do not download, convert, or mutate data by default. Registry inspection and module import are safe; calling a dataset builder,
train_utils.get_dataset, or TFDSdownload_and_prepare()can touch external data. - Resolve unknown dataset names in two phases: first check the lazy registry names, then decide whether the config uses a project/custom dataset whose registration module must be imported or whether
dataset_nameis a typo. - Preflight before COCO/TFRecord conversion requests. If no data is present, provide the required layout, annotation, dependency, and output-path checklist; route actual conversion tools to
baselines-and-projects. - Keep launch mechanics separate. For full train/eval command construction, config flag plumbing, checkpoint/workdir handling, or distributed launches, use
running-and-trainingafter the dataset facts are known. - Keep model-consumption semantics separate. For how models interpret sharded batches, multi-label targets, task heads, or input specs, use
modeling-and-layersafter identifying the batch format here.
04 同仓库更多 Skills
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
python-api
Programmatic Object-Detection-Metrics API guidance for BoundingBox, BoundingBoxes, Evaluator, enums, coordinate conversion, AP/mAP semantics, and source-style import diagnostics.
file-evaluation
Folder-based PASCAL VOC AP/mAP evaluation from ground-truth and detection text files.
object-detection-metrics
Guides legacy Object-Detection-Metrics PASCAL VOC AP/mAP evaluation from detection text folders and direct Python API classes.
nx-workspace-patterns
Configure and optimize Nx monorepo workspaces. Use when setting up Nx, configuring project boundaries, optimizing build caching, or implementing affected commands.