data-preparation
Prepare, validate, and visualize HRM ARC, Sudoku, and Maze puzzle datasets in the repository's shared NumPy layout.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-08
- Stars
- ★ 11
- Forks
- 3
01 前言 Front Matter
- name
- data-preparation
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Prepare, validate, and visualize HRM ARC, Sudoku, and Maze puzzle datasets in the repository's shared NumPy layout.
- disable-model-invocation
- true
02 资源文件 3
- 参考文档references/data-formats.md3.9 KB
- 参考文档references/troubleshooting.md3.2 KB
- 参考文档references/workflows.md3.2 KB
03 SKILL.md 正文
HRM Data Preparation
Use this sub-skill when the task is to create, inspect, validate, or visualize
Hierarchical Reasoning Model (HRM) datasets for ARC, Sudoku, or Maze workflows.
The original repo builders produce a shared directory layout consumed by
PuzzleDataset and by pretrain.py/evaluate.py.
When to use
- The user asks to build ARC-AGI, ARC-AGI-2, ConceptARC, Sudoku Extreme, or Maze datasets for HRM.
- The task names
dataset/build_arc_dataset.py,build_sudoku_dataset.py,build_maze_dataset.py,PuzzleDataset,PuzzleDatasetMetadata,identifiers.json, orall__inputs.npystyle files. - A training or evaluation run fails before model execution because dataset
paths, metadata,
.npyarrays, split/subset names, token ranges, or puzzle index arrays are wrong. - The user wants a browser visualizer for converted puzzle folders.
Route map
- Read references/data-formats.md for the exact shared dataset directory structure, metadata fields, array meanings, and validation invariants.
- Read references/workflows.md for ARC, Sudoku, Maze, and visualization command recipes, including network/submodule prerequisites and safe bounded variants.
- Use references/troubleshooting.md when a
builder cannot find raw data, Hugging Face downloads fail, token ranges look
invalid, or
PuzzleDatasetyields no batches. - Run scripts/validate_dataset_layout.py against a converted dataset before long training or evaluation.
- Use scripts/run_dataset_builder.py when a future task needs a wrapper that forwards arguments to an HRM checkout from any current working directory.
- Open scripts/puzzle_visualizer.html in a
browser for local inspection of generated puzzle folders. The required
npyjsasset is bundled underscripts/assets/.
Core workflow
# ARC-1 from initialized ARC-AGI + ConceptARC submodules
python dataset/build_arc_dataset.py
# ARC-2 from ARC-AGI-2 raw data
python dataset/build_arc_dataset.py \
--dataset-dirs dataset/raw-data/ARC-AGI-2/data \
--output-dir data/arc-2-aug-1000
# Sudoku 1K augmented demo
python dataset/build_sudoku_dataset.py \
--output-dir data/sudoku-extreme-1k-aug-1000 \
--subsample-size 1000 \
--num-aug 1000
# Maze 30x30 hard 1K
python dataset/build_maze_dataset.py --output-dir data/maze-30x30-hard-1k
After building, validate the shared layout before training:
python <skill>/sub-skills/data-preparation/scripts/validate_dataset_layout.py \
data/sudoku-extreme-1k-aug-1000 --splits train test
Replace <skill> with the root of this generated skill directory. The helper is
self-contained; it does not need the original HRM checkout unless you are
running the original builder scripts.
Boundaries
- This sub-skill owns dataset builders, converted layout, schema validation, and local visualization.
- Use
model-architecturefor HRM ACT v1 config fields, losses, model class routing, FlashAttention layers, and sparse puzzle embeddings. - Use
training-evaluationforpretrain.py,evaluate.py, checkpoints, W&B, CUDA training, and ARC prediction post-processing. - Do not run full dataset downloads or raw-data submodule cloning unless the user has accepted network and storage costs. Use tiny fixtures and the bundled validator for bounded checks.
Validation expectations
- Builder
--helpcommands are safe and should list the documented options. - Full Sudoku and Maze conversion require network access to Hugging Face datasets.
- Full ARC conversion requires the ARC/ConceptARC raw-data submodules or an equivalent local directory.
- Converted train/test split directories must contain
dataset.jsonplus<subset>__inputs.npy,<subset>__labels.npy,<subset>__puzzle_identifiers.npy,<subset>__puzzle_indices.npy, and<subset>__group_indices.npyfor each subset in metadatasets.
04 同仓库更多 Skills
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
python-api
Programmatic Object-Detection-Metrics API guidance for BoundingBox, BoundingBoxes, Evaluator, enums, coordinate conversion, AP/mAP semantics, and source-style import diagnostics.
file-evaluation
Folder-based PASCAL VOC AP/mAP evaluation from ground-truth and detection text files.
object-detection-metrics
Guides legacy Object-Detection-Metrics PASCAL VOC AP/mAP evaluation from detection text folders and direct Python API classes.
nx-workspace-patterns
Configure and optimize Nx monorepo workspaces. Use when setting up Nx, configuring project boundaries, optimizing build caching, or implementing affected commands.