create-datasets
Routes Petastorm dataset creation, schema design, metadata repair, copy-and-filter workflows, and row-group indexing from Spark and existing Parquet data.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- create-datasets
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Routes Petastorm dataset creation, schema design, metadata repair, copy-and-filter workflows, and row-group indexing from Spark and existing Parquet data.
- disable-model-invocation
- true
02 资源文件 8
- 参考文档references/api-reference.md4.8 KB
- 参考文档references/cli-reference.md2.2 KB
- 参考文档references/data-formats.md2.3 KB
- 参考文档references/troubleshooting.md2.5 KB
- 参考文档references/workflows.md4.1 KB
- 脚本scripts/smoke_copy_dataset.py3.6 KB
- 脚本scripts/smoke_generate_metadata.py3.3 KB
- 脚本scripts/smoke_make_minimal_dataset.py3.2 KB
03 SKILL.md 正文
Create Datasets
Use this route when the task starts from raw rows, a Spark DataFrame, or an existing Parquet store that needs Petastorm metadata, copying, filtering, schema repair, or row-group indexing.
Typical triggers
- "How do I create a Petastorm dataset?"
- "How do I define a schema or codec?"
- "How do I copy and filter an existing dataset?"
- "How do I regenerate Petastorm metadata?"
- "How do I build row-group indexes?"
- "How do I use the copy or metadata console tools?"
What belongs here
UnischemaandUnischemaFieldScalarCodec,NdarrayCodec,CompressedNdarrayCodec,CompressedImageCodecdict_to_spark_rowmaterialize_datasetget_schemaandget_schema_from_dataset_urlcopy_datasetgenerate_petastorm_metadatabuild_rowgroup_indexSingleFieldIndexerandFieldNotNullIndexer- filesystem URL resolution used while writing or repairing datasets
- the
petastorm-copy-dataset.pyandpetastorm-generate-metadata.pycommands
What does not belong here
- consuming an already-created dataset
- TensorFlow and PyTorch adapter details beyond writer-side shape or schema concerns
- Spark DataFrame caching into reusable loaders
Use sub-skills/read-datasets/ for reader work.
Use sub-skills/spark-converter/ when the task is caching a Spark DataFrame for later loading.
Read this first
references/workflows.mdfor writing, copying, metadata repair, and indexing recipesreferences/api-reference.mdfor verified signatures and behaviorreferences/cli-reference.mdfor the dataset copy and metadata commandsreferences/data-formats.mdfor the on-disk layout and metadata keysreferences/troubleshooting.mdfor schema, Spark, URL, and metadata failuresscripts/smoke_make_minimal_dataset.pyfor a tiny write smoke testscripts/smoke_copy_dataset.pyfor a copy-and-filter smoke testscripts/smoke_generate_metadata.pyfor a metadata-repair smoke test
How to choose a creation path
- Starting from rows or a DataFrame?
- Define a
Unischemaand write throughmaterialize_dataset.
- Define a
- Need to rewrite an existing dataset?
- Use
copy_datasetfor subset/filter/repartition jobs.
- Use
- Need to restore reader compatibility?
- Use
generate_petastorm_metadata.
- Use
- Need row-group acceleration?
- Use
build_rowgroup_indexwith the matching indexer type.
- Use
Boundary reminders
- Non-scalar fields need a codec.
materialize_datasetis the writer-side contract that stamps Petastorm metadata onto the Parquet output.copy_datasetcan narrow columns and drop rows with nulls in selected fields.generate_petastorm_metadatacan infer the schema from the dataset when metadata already exists, or use a supplied unischema class string.build_rowgroup_indexexpects compatible indexers and a writable dataset metadata location.
Common tasks
Create a new dataset
- Define a schema with
Unischema. - Convert dictionaries into Spark rows with
dict_to_spark_row. - Wrap the write in
materialize_dataset. - Write to a
file://,hdfs://,s3://, or compatible URL.
Repair or regenerate metadata
- Open the existing dataset URL.
- Call
generate_petastorm_metadataor the CLI wrapper. - Reopen the dataset with
make_readerto confirm the repair.
Copy or filter a dataset
- Decide which columns should survive the copy.
- Decide which fields must remain non-null.
- Optionally repartition before write.
- Read the result back to confirm the shape.
Build a row-group index
- Create an indexer object for the field you want to accelerate.
- Call
build_rowgroup_indexfrom a Spark context. - Use the resulting selector on the read side.
When to stop and switch routes
- If the user only wants to open the data, switch to
read-datasets. - If the user wants to convert a Spark DataFrame into a reusable cached loader, switch to
spark-converter.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).