AGENT SKILL

create-datasets

Routes Petastorm dataset creation, schema design, metadata repair, copy-and-filter workflows, and row-group indexing from Spark and existing Parquet data.

License 许可
Apache-2.0
最近更新
2026-09-03
Stars
★ 351
Forks
27

01 前言 Front Matter

name
create-datasets
license
Apache 2.0
metadata
{ "disco-role": "operating" }
description
Routes Petastorm dataset creation, schema design, metadata repair, copy-and-filter workflows, and row-group indexing from Spark and existing Parquet data.
disable-model-invocation
true

02 资源文件 8

03 SKILL.md 正文

Create Datasets

Use this route when the task starts from raw rows, a Spark DataFrame, or an existing Parquet store that needs Petastorm metadata, copying, filtering, schema repair, or row-group indexing.

Typical triggers

  • "How do I create a Petastorm dataset?"
  • "How do I define a schema or codec?"
  • "How do I copy and filter an existing dataset?"
  • "How do I regenerate Petastorm metadata?"
  • "How do I build row-group indexes?"
  • "How do I use the copy or metadata console tools?"

What belongs here

  • Unischema and UnischemaField
  • ScalarCodec, NdarrayCodec, CompressedNdarrayCodec, CompressedImageCodec
  • dict_to_spark_row
  • materialize_dataset
  • get_schema and get_schema_from_dataset_url
  • copy_dataset
  • generate_petastorm_metadata
  • build_rowgroup_index
  • SingleFieldIndexer and FieldNotNullIndexer
  • filesystem URL resolution used while writing or repairing datasets
  • the petastorm-copy-dataset.py and petastorm-generate-metadata.py commands

What does not belong here

  • consuming an already-created dataset
  • TensorFlow and PyTorch adapter details beyond writer-side shape or schema concerns
  • Spark DataFrame caching into reusable loaders

Use sub-skills/read-datasets/ for reader work. Use sub-skills/spark-converter/ when the task is caching a Spark DataFrame for later loading.

Read this first

  • references/workflows.md for writing, copying, metadata repair, and indexing recipes
  • references/api-reference.md for verified signatures and behavior
  • references/cli-reference.md for the dataset copy and metadata commands
  • references/data-formats.md for the on-disk layout and metadata keys
  • references/troubleshooting.md for schema, Spark, URL, and metadata failures
  • scripts/smoke_make_minimal_dataset.py for a tiny write smoke test
  • scripts/smoke_copy_dataset.py for a copy-and-filter smoke test
  • scripts/smoke_generate_metadata.py for a metadata-repair smoke test

How to choose a creation path

  1. Starting from rows or a DataFrame?
    • Define a Unischema and write through materialize_dataset.
  2. Need to rewrite an existing dataset?
    • Use copy_dataset for subset/filter/repartition jobs.
  3. Need to restore reader compatibility?
    • Use generate_petastorm_metadata.
  4. Need row-group acceleration?
    • Use build_rowgroup_index with the matching indexer type.

Boundary reminders

  • Non-scalar fields need a codec.
  • materialize_dataset is the writer-side contract that stamps Petastorm metadata onto the Parquet output.
  • copy_dataset can narrow columns and drop rows with nulls in selected fields.
  • generate_petastorm_metadata can infer the schema from the dataset when metadata already exists, or use a supplied unischema class string.
  • build_rowgroup_index expects compatible indexers and a writable dataset metadata location.

Common tasks

Create a new dataset

  1. Define a schema with Unischema.
  2. Convert dictionaries into Spark rows with dict_to_spark_row.
  3. Wrap the write in materialize_dataset.
  4. Write to a file://, hdfs://, s3://, or compatible URL.

Repair or regenerate metadata

  1. Open the existing dataset URL.
  2. Call generate_petastorm_metadata or the CLI wrapper.
  3. Reopen the dataset with make_reader to confirm the repair.

Copy or filter a dataset

  1. Decide which columns should survive the copy.
  2. Decide which fields must remain non-null.
  3. Optionally repartition before write.
  4. Read the result back to confirm the shape.

Build a row-group index

  1. Create an indexer object for the field you want to accelerate.
  2. Call build_rowgroup_index from a Spark context.
  3. Use the resulting selector on the read side.

When to stop and switch routes

  • If the user only wants to open the data, switch to read-datasets.
  • If the user wants to convert a Spark DataFrame into a reusable cached loader, switch to spark-converter.

04 同仓库更多 Skills

VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-triton-opt

Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-solve

END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-skill

PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-pattern-fusion

Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-orchestrate

Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-feedback

Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).

★ 351