datasets-library
Manage ML datasets with the datasets ecosystem — loading, streaming, preprocessing, versioning, and sharing. Use when data handling (not modeling) is the bottleneck.
- License 许可
- MIT
- 最近更新
- 2026-11-07
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- datasets-library
- category
- ai-research
- description
- Manage ML datasets with the datasets ecosystem — loading, streaming, preprocessing, versioning, and sharing. Use when data handling (not modeling) is the bottleneck.
03 SKILL.md 正文
Datasets Library
Models get the glory; datasets decide the outcome. The datasets ecosystem gives you: one-line loading of thousands of public datasets, memory-mapped access to data bigger than RAM, streaming for internet-scale corpora, and versioned, shareable dataset artifacts.
Overview
The core abstractions: Dataset (an Arrow-backed table — fast, memory-mapped, sliceable),
DatasetDict (train/validation/test splits together), and streaming mode (iterate without
downloading). Processing is functional: map for transforms, filter for selection, shuffle,
select, train_test_split — all returning new dataset objects, lazily evaluated and cached.
Push to the hub to version and share.
When to use
- Loading public datasets for training or evaluation without wrestling with formats.
- Preprocessing large corpora: tokenization, filtering, deduplication at scale.
- Streaming datasets too large to download.
- Versioning and sharing your own datasets reproducibly.
Core concepts
- Arrow backing: columnar, memory-mapped storage — datasets larger than RAM are sliceable without loading. Understand this and stop worrying about size.
- Lazy maps:
mapwith batching and multiprocessing, cached to disk. Write the transform once; it runs in parallel and caches. - Streaming: iterate over massive datasets without downloading — for exploration and single-pass training. (Shuffling is approximate in streaming; know the trade-off.)
- Splits: train/validation/test as a DatasetDict; stratified splits for imbalanced data; deterministic seeds for reproducibility.
- Features schema: typed columns (including nested structures, images, audio). Define the schema; it validates your data and documents it.
- Versioning: datasets versioned like code — push to the hub with a revision, pin it in training configs. "Which data trained this model?" should always have an answer.
Practical workflow
- Load and inspect: look at examples, check the features schema, verify splits and sizes.
- Write preprocessing as batched
mapfunctions: tokenization, cleaning, filtering. Cache the result. - Validate the processed data: sample outputs, check label distributions, verify no leakage between splits.
- For huge data: switch to streaming for exploration; materialize only what's needed.
- Version the final dataset: push with a clear name, README (documenting source, processing, license), and pinned revision.
- Reference the pinned revision in every training run — data version is part of the experiment.
Dataset pipeline:
load → inspect (examples, schema, splits)
→ map (tokenize/clean, batched, cached)
→ filter (quality, dedupe)
→ validate (samples, distributions, leakage check)
→ split (deterministic) → version (push + pin)
Common pitfalls
- Uninspected data: training on a dataset you never looked at. Sample everything; surprises are the norm.
- Leakage between splits: duplicates or near-duplicates across train/test. Deduplicate globally before splitting.
- Unversioned data: "the dataset" changing under you. Pin revisions; treat data as code.
- Streaming shuffle naivety: assuming streaming shuffles perfectly. It approximates — buffer size matters.
- License blindness: public dataset ≠ free to use commercially. Check licenses before training products on data.
- Preprocessing in training code: tokenization buried in the training script, uncached, rerun every time. Preprocess once, cache, version.
04 同仓库更多 Skills
web-scrape
Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.
design-taste-frontend
tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.
test-driven-development
Use when implementing any feature or bugfix, before writing implementation code
systematic-debugging
Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
requesting-code-review
Use when completing tasks, implementing major features, or before merging to verify work meets requirements
brainstorming
You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).