AGENT SKILL

datasets-library

Manage ML datasets with the datasets ecosystem — loading, streaming, preprocessing, versioning, and sharing. Use when data handling (not modeling) is the bottleneck.

License 许可
MIT
所属仓库
TanNhatCMS/jarvis
最近更新
2026-11-07
Stars
★ 0
Forks
1

01 前言 Front Matter

name
datasets-library
category
ai-research
description
Manage ML datasets with the datasets ecosystem — loading, streaming, preprocessing, versioning, and sharing. Use when data handling (not modeling) is the bottleneck.

03 SKILL.md 正文

Datasets Library

Models get the glory; datasets decide the outcome. The datasets ecosystem gives you: one-line loading of thousands of public datasets, memory-mapped access to data bigger than RAM, streaming for internet-scale corpora, and versioned, shareable dataset artifacts.

Overview

The core abstractions: Dataset (an Arrow-backed table — fast, memory-mapped, sliceable), DatasetDict (train/validation/test splits together), and streaming mode (iterate without downloading). Processing is functional: map for transforms, filter for selection, shuffle, select, train_test_split — all returning new dataset objects, lazily evaluated and cached. Push to the hub to version and share.

When to use

  • Loading public datasets for training or evaluation without wrestling with formats.
  • Preprocessing large corpora: tokenization, filtering, deduplication at scale.
  • Streaming datasets too large to download.
  • Versioning and sharing your own datasets reproducibly.

Core concepts

  • Arrow backing: columnar, memory-mapped storage — datasets larger than RAM are sliceable without loading. Understand this and stop worrying about size.
  • Lazy maps: map with batching and multiprocessing, cached to disk. Write the transform once; it runs in parallel and caches.
  • Streaming: iterate over massive datasets without downloading — for exploration and single-pass training. (Shuffling is approximate in streaming; know the trade-off.)
  • Splits: train/validation/test as a DatasetDict; stratified splits for imbalanced data; deterministic seeds for reproducibility.
  • Features schema: typed columns (including nested structures, images, audio). Define the schema; it validates your data and documents it.
  • Versioning: datasets versioned like code — push to the hub with a revision, pin it in training configs. "Which data trained this model?" should always have an answer.

Practical workflow

  1. Load and inspect: look at examples, check the features schema, verify splits and sizes.
  2. Write preprocessing as batched map functions: tokenization, cleaning, filtering. Cache the result.
  3. Validate the processed data: sample outputs, check label distributions, verify no leakage between splits.
  4. For huge data: switch to streaming for exploration; materialize only what's needed.
  5. Version the final dataset: push with a clear name, README (documenting source, processing, license), and pinned revision.
  6. Reference the pinned revision in every training run — data version is part of the experiment.
Dataset pipeline:
load → inspect (examples, schema, splits)
→ map (tokenize/clean, batched, cached)
→ filter (quality, dedupe)
→ validate (samples, distributions, leakage check)
→ split (deterministic) → version (push + pin)

Common pitfalls

  • Uninspected data: training on a dataset you never looked at. Sample everything; surprises are the norm.
  • Leakage between splits: duplicates or near-duplicates across train/test. Deduplicate globally before splitting.
  • Unversioned data: "the dataset" changing under you. Pin revisions; treat data as code.
  • Streaming shuffle naivety: assuming streaming shuffles perfectly. It approximates — buffer size matters.
  • License blindness: public dataset ≠ free to use commercially. Check licenses before training products on data.
  • Preprocessing in training code: tokenization buried in the training script, uncached, rerun every time. Preprocess once, cache, version.

04 同仓库更多 Skills

TanNhatCMS
TanNhatCMS/jarvis

web-scrape

Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

design-taste-frontend

tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

test-driven-development

Use when implementing any feature or bugfix, before writing implementation code

★ 0
TanNhatCMS
TanNhatCMS/jarvis

systematic-debugging

Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes

★ 0
TanNhatCMS
TanNhatCMS/jarvis

requesting-code-review

Use when completing tasks, implementing major features, or before merging to verify work meets requirements

★ 0
TanNhatCMS
TanNhatCMS/jarvis

brainstorming

You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).

★ 0