AGENT SKILL

dataset-writing

Convert indexable datasets or already-available WebDataset shards into validated FFCV .beton files, with ordered fields, writer controls, built-in/custom fields, and read-back troubleshooting.

License 许可
Apache-2.0
最近更新
2026-09-03
Stars
★ 351
Forks
27

01 前言 Front Matter

name
dataset-writing
license
Apache 2.0
metadata
{ "disco-role": "operating" }
description
Convert indexable datasets or already-available WebDataset shards into validated FFCV .beton files, with ordered fields, writer controls, built-in/custom fields, and read-back troubleshooting.
disable-model-invocation
true

02 资源文件 3

03 SKILL.md 正文

FFCV dataset writing

Use this sub-skill when a Researcher must serialize an indexable dataset, or already-available local WebDataset shards, into an FFCV .beton file. It owns DatasetWriter, field mapping/order, writer resource controls, built-in and custom field encoding, and validation of the resulting file. It does not own training-time loader pipelines, augmentation design, or obtaining shards.

Operating contract

Inputs

  • A destination .beton path. DatasetWriter.prepare() opens it in wb mode, so treat an existing destination as replaceable output.
  • Either an object with __len__ and __getitem__, or a list of local shard paths plus a worker-safe WebDataset pipeline.
  • A mapping of short ASCII field names to ffcv.fields.Field instances. Python mapping insertion order is the sample serialization order.
  • A source adapter whose returned tuple/list has exactly the same positions as the field mapping. The writer zips values and does not provide a reliable schema/arity check for you.
  • Optional indexed-dataset controls: indices, shuffle_indices, and positive chunksize; optional writer controls: page_size and num_workers.

Outputs and acceptance observations

  • from_indexed_dataset(...) or, when the optional dependency is installed, from_webdataset(...) completes without a worker exception.
  • Reader(path) opens the file, the sample count and logical field names match the intended output, and fixed metadata or decoded payloads match a source check. A path merely existing is not success.
  • Variable-length bytes/JSON are checked with their length/terminator contract, not compared as if every batch had the original unpadded length.
  • A custom type-255 field is opened with the required reader registration and its decoder returns the intended representation.

Non-negotiable rules

  1. DatasetWriter pairs fields.values() with each sample by position. A mapping's insertion order is useful for the field mapping, but a sample dict iterates its keys, not its values. Convert dict samples explicitly to a tuple in the same named order.
  2. Use unique ASCII names no longer than 16 bytes. The on-disk descriptor has a 16-byte name slot and this build truncates longer names rather than rejecting them; truncation can collide or make a field inaccessible.
  3. NDArrayField and TorchTensorField are fixed shape/dtype fields. BytesField and JSONField allocate variable-size payloads. The normal bytes decoder allocates each decoded batch to its largest item, so preserve lengths or use a delimiter/terminator protocol when exact trimming matters.
  4. JSONField writes UTF-8 JSON followed by a NUL byte. Decode with BytesDecoder and call JSONField.unpack; do not pass padded bytes directly to json.loads.
  5. A non-exact built-in Field subclass is written with type id 255. The writer has no custom registry; the reader needs a field-name-to-class registration (Loader(custom_fields=...) or Reader(custom_handlers=...)).
  6. WebDataset is optional and local-only here. from_webdataset imports it on demand, counts each shard before writing, and then processes shards in workers. No network or download procedure belongs in this skill.

Fast route

  1. Inspect a small, typical, and largest/most variable sample in the parent process. Check tuple arity, scalar types, array shape/dtype, image layout, byte-array format, and JSON encodability.
  2. Define a short ordered mapping and, for dict-shaped data, an explicit FIELD_ORDER adapter. Start with a tiny subset and num_workers=1.
  3. Use the recipes in references/workflows.md for indexed conversion, indices/shuffle/chunk controls, local WebDataset conversion, structural read-back, JSON unpacking, and custom registration.
  4. Choose writer resources only after the smoke file passes. page_size is a power of two in [1 << 21, 1 << 32); one allocation must fit in one page. num_workers=-1 resolves to the process CPU affinity, and chunksize controls indexed work-queue granularity.
  5. For field-specific input and serialization rules, use references/fields-and-formats.md.
  6. For failures, reduce to one worker and a small failing index, then use references/troubleshooting.md. Preserve the smallest failing sample and the exact field mapping.

Built-in field selection

  • IntField and FloatField: fixed scalar integer/float metadata (int64 and float64 in this build).
  • NDArrayField(dtype, shape): fixed-size NumPy arrays; ragged rows need a deliberate padding scheme or BytesField plus a length contract.
  • TorchTensorField(torch_dtype, shape): fixed-size CPU torch tensors whose .numpy() conversion succeeds; normalize CUDA or grad-tracking tensors in the source adapter before writing.
  • RGBImageField: HWC RGB uint8 NumPy arrays or PIL images, with raw/JPEG, smart/proportion, and optional max-resolution preprocessing. Normalize torch images to CPU HWC uint8; this writer implementation does not accept a CHW torch tensor directly.
  • BytesField: one-dimensional NumPy uint8 arrays. Convert Python bytes explicitly with np.frombuffer and define how the original length is kept.
  • JSONField: JSON-compatible Python values, usually dicts; use its NUL-aware unpack helper after a BytesDecoder read-back.

Evidence and verified baseline

This operating graph was distilled from setup.py, ffcv/writer.py, ffcv/reader.py, the field implementations, docs/writing_datasets.rst, docs/quickstart.rst, and the requested writer/custom-field/JSON/WebDataset tests. The inspected environment reported distribution metadata 1.0.1, ffcv.__version__ 1.0.2, Python 3.11, a passing CPU smoke conversion, and a successful compiled-extension import. Treat the distribution/module version split as a provenance warning: recreate and read files with the same installed build rather than assuming labels alone prove format compatibility.

No bundled executable script is included. A generic writer script would have to guess the user's schema, source lifetime, output replacement policy, optional WebDataset availability, page size, worker count, and custom-field classes; its multiprocessing and disk allocation side effects make reference-only recipes safer for this sub-skill.

04 同仓库更多 Skills

VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-triton-opt

Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-solve

END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-skill

PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-pattern-fusion

Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-orchestrate

Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-feedback

Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).

★ 351