data-and-tracking
Use DALLE2-pytorch dataloaders, WebDataset and embedding layouts, tracker/log/save/load configuration, and safe data-layout diagnostics for training workflows.
- License 许可
- MIT
- 最近更新
- 2026-10-08
- Stars
- ★ 11
- Forks
- 3
01 前言 Front Matter
- name
- data-and-tracking
- license
- MIT
- metadata
- { "disco-role": "operating" }
- description
- Use DALLE2-pytorch dataloaders, WebDataset and embedding layouts, tracker/log/save/load configuration, and safe data-layout diagnostics for training workflows.
- disable-model-invocation
- true
02 资源文件 3
- 参考文档references/data-formats.md4.9 KB
- 参考文档references/tracking-and-checkpoints.md4.0 KB
- 参考文档references/troubleshooting.md3.5 KB
03 SKILL.md 正文
data-and-tracking
Use this sub-skill when the task is to prepare DALLE2-pytorch training data, validate WebDataset/embedding shard layout, configure ImageEmbeddingDataset or PriorEmbeddingDataset, or set up experiment trackers, checkpoint loading, and checkpoint saving.
Load This When
- Checking decoder WebDataset
.tarshards,.jpg/.npykeys, sidecar image/text embedding folders, shard widths, orindex_width. - Using
create_image_embedding_dataloader,ImageEmbeddingDataset,get_reader,make_splits, orPriorEmbeddingDataset. - Debugging S3/fsspec data URLs,
s3cmd,s3fs, missing embeddings, zero sidecar embeddings, or resampling/shuffling behavior. - Configuring console/W&B logging, local/URL/W&B checkpoint loading, or local/W&B/HuggingFace saving.
- Separating public package workflow guidance from credentials, private data, and tracker artifacts.
Route Elsewhere
- Model architecture, prior/decoder construction,
DALLE2,dream, inpainting, or CLIP adapters:../generation-and-api/SKILL.md. - JSON training configs, trainer APIs, launcher commands, and distributed training:
../training-and-configs/SKILL.md.
Runtime Contract
- Public package:
pip install dalle2-pytorch. - Public data imports:
from dalle2_pytorch.dataloaders import ImageEmbeddingDataset, create_image_embedding_dataloader, get_reader, make_splits, PriorEmbeddingDataset. - Public tracker/config imports:
from dalle2_pytorch.train_configs import TrackerConfig; tracker provider behavior lives behind config classes. - This skill does not provide datasets, credentials, model weights, S3 buckets, W&B runs, or HuggingFace tokens. It helps validate user-provided paths/configs safely.
References And Bundled Script
- Decoder and prior data formats: references/data-formats.md.
- Tracker, loader, saver, and checkpoint destinations: references/tracking-and-checkpoints.md.
- Data/tracker troubleshooting: references/troubleshooting.md.
- Safe layout checker: scripts/validate_webdataset_layout.py.
Typical Flow
-
Read references/data-formats.md to decide whether the task uses decoder WebDataset shards or prior EmbeddingReader folders.
-
For decoder configs, inspect shard and sidecar naming without loading large data:
python scripts/validate_webdataset_layout.py --tar-pattern 'data/shards/{}.tar' --start-shard 0 --end-shard 9 --shard-width 1 --index-width 4 -
Add sidecar folders when embeddings are outside the tar files:
python scripts/validate_webdataset_layout.py --tar-pattern 'data/shards/{}.tar' --start-shard 0 --end-shard 9 --shard-width 1 --index-width 4 --image-embeddings data/img_emb --text-embeddings data/text_emb -
Read references/tracking-and-checkpoints.md before enabling W&B, HuggingFace, S3, URL loaders, or auto-resume.
Safety Defaults
- Prefer local filesystem paths and
log_type: consolefor first checks. - Do not inspect or upload private tracker credentials; require the user to supply and approve them at execution time.
- Treat network, S3, W&B, HuggingFace, and URL loaders/savers as credentialed or networked workflows, not default smoke tests.
- Use
--inspect-taron at most a small shard; otherwise the bundled checker only validates names and file existence.
04 同仓库更多 Skills
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
python-api
Programmatic Object-Detection-Metrics API guidance for BoundingBox, BoundingBoxes, Evaluator, enums, coordinate conversion, AP/mAP semantics, and source-style import diagnostics.
file-evaluation
Folder-based PASCAL VOC AP/mAP evaluation from ground-truth and detection text files.
object-detection-metrics
Guides legacy Object-Detection-Metrics PASCAL VOC AP/mAP evaluation from detection text folders and direct Python API classes.
nx-workspace-patterns
Configure and optimize Nx monorepo workspaces. Use when setting up Nx, configuring project boundaries, optimizing build caching, or implementing affected commands.